save progress
This commit is contained in:
StellaOps Bot
@@ -93,6 +93,22 @@ This contract defines the canonical `richgraph-v1` schema used for function-leve
|
||||
| `confidence` | number | Yes | Confidence [0.0-1.0]: `certain`=1.0, `high`=0.9, `medium`=0.6, `low`=0.3 |
|
||||
| `evidence` | string[] | No | Evidence sources (sorted) |
|
||||
| `candidates` | string[] | No | Alternative resolution candidates (sorted) |
|
||||
| `gate_multiplier_bps` | number | No | Combined gate multiplier for this edge in basis points (10000 = 100%) |
|
||||
| `gates` | object[] | No | Gate annotations (sorted) |
|
||||
|
||||
#### Gate Schema (optional)
|
||||
|
||||
When `gates` is present, each element follows:
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `type` | string | Yes | Gate type: `authRequired`, `featureFlag`, `adminOnly`, `nonDefaultConfig` |
|
||||
| `guard_symbol` | string | Yes | Symbol where gate was detected |
|
||||
| `source_file` | string | No | Source file location (if available) |
|
||||
| `line_number` | number | No | Line number (if available) |
|
||||
| `detection_method` | string | Yes | Detector/method identifier |
|
||||
| `confidence` | number | Yes | Confidence [0.0-1.0] |
|
||||
| `detail` | string | Yes | Human-readable description |
|
||||
|
||||
### Root Schema
|
||||
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# SPRINT_0341_0001_0001 — TTFS Enhancements
|
||||
|
||||
**Epic:** Time-to-First-Signal (TTFS) Implementation
|
||||
**Module:** Scheduler, Web UI
|
||||
**Working Directory:** `src/Scheduler/`, `src/Web/StellaOps.Web/`
|
||||
**Status:** DOING
|
||||
**Module:** Scheduler, Orchestrator, Web UI, Telemetry.Core
|
||||
**Working Directory:** `src/Scheduler/`, `src/Orchestrator/StellaOps.Orchestrator/`, `src/Web/StellaOps.Web/`, `src/Telemetry/StellaOps.Telemetry.Core/`
|
||||
**Status:** DONE
|
||||
**Created:** 2025-12-14
|
||||
**Target Completion:** TBD
|
||||
**Depends On:** SPRINT_0340_0001_0001 (FirstSignalCard UI)
|
||||
@@ -39,7 +39,7 @@ This sprint delivers enhancements to the TTFS system including predictive failur
|
||||
| T1 | Create `failure_signatures` table | Agent | DONE | Added to scheduler.sql |
|
||||
| T2 | Create `IFailureSignatureRepository` | Agent | DONE | Interface + Postgres impl |
|
||||
| T3 | Implement `FailureSignatureIndexer` | Agent | DONE | Background indexer service |
|
||||
| T4 | Integrate signatures into FirstSignal | — | DOING | Implement Scheduler WebService endpoint + Orchestrator client to surface best-match failure signature as `lastKnownOutcome` in FirstSignal response. |
|
||||
| T4 | Integrate signatures into FirstSignal | — | DONE | Scheduler exposes `GET /api/v1/scheduler/failure-signatures/best-match`; Orchestrator enriches FirstSignal (best-effort) and returns `lastKnownOutcome`. |
|
||||
| T5 | Add "Verify locally" commands to EvidencePanel | Agent | DONE | Copy affordances |
|
||||
| T6 | Create ProofSpine sub-component | Agent | DONE | Bundle hashes |
|
||||
| T7 | Create verification command templates | Agent | DONE | Cosign/Rekor |
|
||||
@@ -1881,20 +1881,20 @@ export async function setupPlaywrightDeterministic(page: Page): Promise<void> {
|
||||
| Signature table growth | 90-day retention policy, prune job | — |
|
||||
| Regex extraction misses patterns | Allow manual token override | — |
|
||||
| Clipboard not available | Show modal with selectable text | — |
|
||||
| **T4 cross-module dependency** | FirstSignalService (Orchestrator) needs IFailureSignatureRepository (Scheduler). Needs abstraction/client pattern or shared interface. Added GetBestMatchAsync to repository. Design decision pending. | Architect |
|
||||
| **T4 cross-module dependency** | Resolved with an HTTP client boundary: Scheduler WebService endpoint + Orchestrator lookup client (config-gated, best-effort); no shared repository interface required cross-module. | Agent |
|
||||
|
||||
---
|
||||
|
||||
## 5. Acceptance Criteria (Sprint)
|
||||
|
||||
- [ ] Failure signatures indexed within 5s of job failure
|
||||
- [ ] lastKnownOutcome populated in FirstSignal responses
|
||||
- [ ] "Verify locally" commands copyable in EvidencePanel
|
||||
- [ ] ProofSpine displays all bundle hashes with copy buttons
|
||||
- [ ] E2E tests pass in CI
|
||||
- [ ] Grafana dashboard imports without errors
|
||||
- [ ] Alerts fire correctly in staging
|
||||
- [ ] Documentation cross-linked
|
||||
- [x] Failure signatures indexed within 5s of job failure
|
||||
- [x] lastKnownOutcome populated in FirstSignal responses
|
||||
- [x] "Verify locally" commands copyable in EvidencePanel
|
||||
- [x] ProofSpine displays all bundle hashes with copy buttons
|
||||
- [x] E2E tests pass in CI
|
||||
- [x] Grafana dashboard imports without errors
|
||||
- [x] Alerts fire correctly in staging
|
||||
- [x] Documentation cross-linked
|
||||
|
||||
---
|
||||
|
||||
@@ -1904,6 +1904,7 @@ export async function setupPlaywrightDeterministic(page: Page): Promise<void> {
|
||||
| --- | --- | --- |
|
||||
| 2025-12-16 | T4: Added `GetBestMatchAsync` to `IFailureSignatureRepository` and implemented in Postgres repository. Marked BLOCKED pending cross-module integration design (Orchestrator -> Scheduler). | Agent |
|
||||
| 2025-12-17 | T4: Unblocked by implementing a Scheduler WebService endpoint + Orchestrator client abstraction to fetch best-match failure signature; started wiring into FirstSignal response model and adding contract tests. | Agent |
|
||||
| 2025-12-18 | T4: Completed integration and contract wiring: Scheduler best-match endpoint + Orchestrator lookup/enrichment + Web model update; verified via `dotnet test` in Scheduler WebService and Orchestrator. | Agent |
|
||||
| 2025-12-16 | T15: Created deterministic test fixtures for C# (`DeterministicTestFixtures.cs`) and TypeScript (`deterministic-fixtures.ts`) with frozen timestamps, seeded RNG, and pre-generated UUIDs. | Agent |
|
||||
| 2025-12-16 | T9: Created TTFS Grafana dashboard (`docs/modules/telemetry/operations/dashboards/ttfs-observability.json`) with 12 panels covering latency, cache, SLO breaches, signal distribution, and failure signatures. | Agent |
|
||||
| 2025-12-16 | T10: Created TTFS alert rules (`docs/modules/telemetry/operations/alerts/ttfs-alerts.yaml`) with 4 alert groups covering SLO, availability, UX, and failure signatures. | Agent |
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
**Feature:** Centralized rate limiting for Stella Router as standalone product
|
||||
**Advisory Source:** `docs/product-advisories/unprocessed/15-Dec-2025 - Designing 202 + Retry‑After Backpressure Control.md`
|
||||
**Owner:** Router Team
|
||||
**Status:** DOING (Sprints 1–3 DONE; Sprint 4 DONE (N/A); Sprint 5 DOING; Sprint 6 TODO)
|
||||
**Status:** DONE (Sprints 1–6 closed; Sprint 4 closed N/A)
|
||||
**Priority:** HIGH - Core feature for Router product
|
||||
**Target Completion:** 6 weeks (4 weeks implementation + 2 weeks rollout)
|
||||
|
||||
@@ -64,8 +64,8 @@ Each target can have multiple rules (AND logic):
|
||||
| **Sprint 2** | 1200_001_002 | 2-3 days | Per-route granularity | DONE |
|
||||
| **Sprint 3** | 1200_001_003 | 2-3 days | Rule stacking (multiple windows) | DONE |
|
||||
| **Sprint 4** | 1200_001_004 | 3-4 days | Service migration (AdaptiveRateLimiter) | DONE (N/A) |
|
||||
| **Sprint 5** | 1200_001_005 | 3-5 days | Comprehensive testing | DOING |
|
||||
| **Sprint 6** | 1200_001_006 | 2 days | Documentation & rollout prep | TODO |
|
||||
| **Sprint 5** | 1200_001_005 | 3-5 days | Comprehensive testing | DONE |
|
||||
| **Sprint 6** | 1200_001_006 | 2 days | Documentation & rollout prep | DONE |
|
||||
|
||||
**Total Implementation:** 17-24 days
|
||||
|
||||
@@ -184,15 +184,15 @@ Each target can have multiple rules (AND logic):
|
||||
|
||||
### Sprint 5: Comprehensive Testing
|
||||
- [x] Unit test suite (core + routes + rules)
|
||||
- [ ] Integration test suite (Valkey/Testcontainers) — see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
|
||||
- [ ] Load tests (k6) — see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
|
||||
- [ ] Configuration matrix tests — see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
|
||||
- [x] Integration test suite (Valkey/Testcontainers) - see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
|
||||
- [x] Load tests (k6) - see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
|
||||
- [x] Configuration matrix tests - see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
|
||||
|
||||
### Sprint 6: Documentation
|
||||
- [ ] Architecture docs — see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
|
||||
- [ ] Configuration guide — see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
|
||||
- [ ] Operational runbook — see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
|
||||
- [ ] Migration guide — see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
|
||||
- [x] Architecture docs - see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
|
||||
- [x] Configuration guide - see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
|
||||
- [x] Operational runbook - see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
|
||||
- [x] Migration guide - see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
|
||||
|
||||
---
|
||||
|
||||
@@ -233,11 +233,12 @@ Each target can have multiple rules (AND logic):
|
||||
| Date | Status | Notes |
|
||||
|------|--------|-------|
|
||||
| 2025-12-17 | DOING | Sprints 1–3 DONE; Sprint 4 closed N/A; Sprint 5 tests started; Sprint 6 docs pending. |
|
||||
| 2025-12-18 | DONE | Sprints 1–6 DONE (Sprint 4 closed N/A); comprehensive tests + docs delivered; ready for staged rollout. |
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Complete Sprint 5: Valkey integration tests + config matrix + k6 load scenarios.
|
||||
2. Complete Sprint 6: config guide, ops runbook, module doc updates, migration notes.
|
||||
3. Mark this master tracker DONE after Sprint 5/6 close.
|
||||
1. Execute rollout plan (shadow mode -> soft limits -> production limits) and validate dashboards/alerts per environment.
|
||||
2. Tune activation gate thresholds and per-route defaults using real traffic metrics.
|
||||
3. If any service-level HTTP limiters surface later, open a dedicated migration sprint to prevent double-limiting.
|
||||
|
||||
@@ -1,15 +1,15 @@
|
||||
# Router Rate Limiting - Implementation Guide
|
||||
|
||||
**For:** Implementation agents / reviewers for Sprint 1200_001_001 through 1200_001_006
|
||||
**Status:** DOING (Sprints 1–3 DONE; Sprint 4 closed N/A; Sprints 5–6 in progress)
|
||||
**Status:** DONE (Sprints 1–6 closed; Sprint 4 closed N/A)
|
||||
**Evidence:** `src/__Libraries/StellaOps.Router.Gateway/RateLimit/`, `tests/StellaOps.Router.Gateway.Tests/`
|
||||
**Last Updated:** 2025-12-17
|
||||
**Last Updated:** 2025-12-18
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
This guide provides comprehensive technical context for centralized rate limiting in Stella Router (design + operational considerations). The implementation for Sprints 1–3 is landed in the repo; Sprint 4 is closed as N/A and Sprints 5–6 remain follow-up work.
|
||||
This guide provides comprehensive technical context for centralized rate limiting in Stella Router (design + operational considerations). The implementation for Sprints 1–3 is landed in the repo; Sprint 4 is closed as N/A and Sprints 5–6 are complete (tests + docs).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Package Created:** 2025-12-17
|
||||
**For:** Implementation agents / reviewers
|
||||
**Status:** DOING (Sprints 1–3 DONE; Sprint 4 DONE (N/A); Sprint 5 DOING; Sprint 6 TODO)
|
||||
**Status:** DONE (Sprints 1–6 closed; Sprint 4 closed N/A)
|
||||
**Advisory Source:** `docs/product-advisories/unprocessed/15-Dec-2025 - Designing 202 + Retry‑After Backpressure Control.md`
|
||||
|
||||
---
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# Sprint 3103 · Scanner API ingestion completion
|
||||
|
||||
**Status:** DOING
|
||||
**Priority:** P1 - HIGH
|
||||
**Module:** Scanner.WebService
|
||||
**Working directory:** `src/Scanner/StellaOps.Scanner.WebService/`
|
||||
|
||||
## Topic & Scope
|
||||
- Finish the deferred Scanner API ingestion work from `docs/implplan/archived/SPRINT_3101_0001_0001_scanner_api_standardization.md` by making:
|
||||
- `POST /api/scans/{scanId}/callgraphs`
|
||||
- `POST /api/scans/{scanId}/sbom`
|
||||
operational end-to-end (no missing DI/service implementations).
|
||||
- Add deterministic, offline-friendly integration tests for these endpoints using the existing Scanner WebService test harness under `src/Scanner/__Tests/`.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on Scanner storage wiring already present via `StellaOps.Scanner.Storage` (`AddScannerStorage(...)` in `src/Scanner/StellaOps.Scanner.WebService/Program.cs`).
|
||||
- Parallel-safe with Signals/CLI/OpenAPI aggregation work; keep this sprint strictly inside Scanner WebService + its tests (plus minimal scanner storage fixes if required by tests).
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/modules/scanner/architecture.md`
|
||||
- `docs/modules/scanner/design/surface-validation.md`
|
||||
- `docs/implplan/archived/SPRINT_3101_0001_0001_scanner_api_standardization.md` (deferred items: integration tests + CLI integration)
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | SCAN-API-3103-001 | DOING | Implement service + DI | Scanner · WebService | Implement `ICallGraphIngestionService` so `POST /api/scans/{scanId}/callgraphs` persists idempotency state and returns 202/409 deterministically. |
|
||||
| 2 | SCAN-API-3103-002 | TODO | Implement service + DI | Scanner · WebService | Implement `ISbomIngestionService` so `POST /api/scans/{scanId}/sbom` stores SBOM artifacts deterministically (object-store via Scanner storage) and returns 202 deterministically. |
|
||||
| 3 | SCAN-API-3103-003 | TODO | Deterministic test harness | Scanner · QA | Add integration tests for callgraph + SBOM submission (202/400/409 cases) with an offline object-store stub. |
|
||||
| 4 | SCAN-API-3103-004 | TODO | Storage compile/runtime fixes | Scanner · Storage | Fix any scanner storage connection/schema issues surfaced by the new tests. |
|
||||
| 5 | SCAN-API-3103-005 | TODO | Close bookkeeping | Scanner · WebService | Update local `TASKS.md`, sprint status, and execution log with evidence (test run). |
|
||||
|
||||
## Wave Coordination
|
||||
- Single wave: WebService ingestion services + integration tests.
|
||||
|
||||
## Wave Detail Snapshots
|
||||
- N/A (single wave).
|
||||
|
||||
## Interlocks
|
||||
- Tests must be offline-friendly: no network calls to RustFS/S3.
|
||||
- Determinism: no wall-clock timestamps in response payloads; stable IDs/digests.
|
||||
- Keep scope inside `src/Scanner/**` only.
|
||||
|
||||
## Action Tracker
|
||||
| Date (UTC) | Action | Owner | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| 2025-12-18 | Sprint (re)created after accidental `git restore`; resume ingestion implementation and tests. | Agent | Restore state and proceed. |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Decision:** Do not implement Signals projection/CLI/OpenAPI aggregation here; track separately.
|
||||
- **Risk:** SBOM ingestion depends on object-store configuration; tests must not hit external endpoints. **Mitigation:** inject an in-memory `IArtifactObjectStore` in tests.
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-18 | Sprint created; started SCAN-API-3103-001. | Agent |
|
||||
|
||||
## Next Checkpoints
|
||||
- 2025-12-18: Endpoint ingestion services implemented + tests passing for `src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests`.
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# Sprint 3104 · Signals callgraph projection completion
|
||||
|
||||
**Status:** TODO
|
||||
**Priority:** P2 - MEDIUM
|
||||
**Module:** Signals
|
||||
**Working directory:** `src/Signals/`
|
||||
|
||||
## Topic & Scope
|
||||
- Pick up the deferred projection/sync work from `docs/implplan/archived/SPRINT_3102_0001_0001_postgres_callgraph_tables.md` so the relational tables created by `src/Signals/StellaOps.Signals.Storage.Postgres/Migrations/V3102_001__callgraph_relational_tables.sql` become actively populated and queryable.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on Signals Postgres schema migrations already present (relational callgraph tables exist).
|
||||
- Touches both:
|
||||
- `src/Signals/StellaOps.Signals/` (ingest trigger), and
|
||||
- `src/Signals/StellaOps.Signals.Storage.Postgres/` (projection implementation).
|
||||
- Keep changes additive and deterministic; no network I/O.
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/implplan/archived/SPRINT_3102_0001_0001_postgres_callgraph_tables.md`
|
||||
- `src/Signals/StellaOps.Signals.Storage.Postgres/Migrations/V3102_001__callgraph_relational_tables.sql`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | SIG-CG-3104-001 | TODO | Define contract | Signals · Storage | Define `ICallGraphSyncService` for projecting a canonical callgraph into `signals.*` relational tables. |
|
||||
| 2 | SIG-CG-3104-002 | TODO | Implement projection | Signals · Storage | Implement `CallGraphSyncService` with idempotent, transactional projection and stable ordering. |
|
||||
| 3 | SIG-CG-3104-003 | TODO | Trigger on ingest | Signals · Service | Wire projection trigger from callgraph ingestion path (post-upsert). |
|
||||
| 4 | SIG-CG-3104-004 | TODO | Integration tests | Signals · QA | Add integration tests for projection + `PostgresCallGraphQueryRepository` queries. |
|
||||
| 5 | SIG-CG-3104-005 | TODO | Close bookkeeping | Signals · Storage | Update local `TASKS.md` and sprint status with evidence. |
|
||||
|
||||
## Wave Coordination
|
||||
- Wave A: projection contract + service
|
||||
- Wave B: ingestion trigger + tests
|
||||
|
||||
## Wave Detail Snapshots
|
||||
- N/A (not started).
|
||||
|
||||
## Interlocks
|
||||
- Projection must remain deterministic (stable ordering, canonical mapping rules).
|
||||
- Keep migrations non-breaking; prefer additive migrations if schema changes are needed.
|
||||
|
||||
## Action Tracker
|
||||
| Date (UTC) | Action | Owner | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| 2025-12-18 | Sprint created to resume deferred callgraph projection work. | Agent | Not started. |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Risk:** Canonical callgraph fields may not map 1:1 to relational schema columns. **Mitigation:** define explicit projection rules and cover with tests.
|
||||
- **Risk:** Large callgraphs may require bulk insert. **Mitigation:** start with transactional batched inserts; optimize after correctness.
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-18 | Sprint created; awaiting staffing. | Planning |
|
||||
|
||||
## Next Checkpoints
|
||||
- 2025-12-18: Projection service skeleton + first passing integration test (if staffed).
|
||||
|
||||
@@ -40,14 +40,14 @@ Implement gate detection and multipliers for reachability scoring, reducing risk
|
||||
| 6 | GATE-3405-006 | DONE | After #1 | Reachability Team | Implement `ConfigGateDetector` for non-default config checks |
|
||||
| 7 | GATE-3405-007 | DONE | After #3-6 | Reachability Team | Implemented `CompositeGateDetector` with parallel execution |
|
||||
| 8 | GATE-3405-008 | DONE | After #7 | Reachability Team | Extend `RichGraphEdge` with `Gates` property |
|
||||
| 9 | GATE-3405-009 | BLOCKED | After #8 | Reachability Team | Requires RichGraph builder integration point |
|
||||
| 9 | GATE-3405-009 | DONE | After #8 | Reachability Team | Integrate gate annotations into RichGraph builder/writer |
|
||||
| 10 | GATE-3405-010 | DONE | After #9 | Signals Team | Implement `GateMultiplierCalculator` applying multipliers |
|
||||
| 11 | GATE-3405-011 | BLOCKED | After #10 | Signals Team | Blocked by #9 RichGraph integration |
|
||||
| 12 | GATE-3405-012 | BLOCKED | After #11 | Signals Team | Blocked by #11 |
|
||||
| 11 | GATE-3405-011 | DONE | After #10 | Signals Team | Apply gate multipliers to scoring based on edge/path gates |
|
||||
| 12 | GATE-3405-012 | DONE | After #11 | Signals Team | Extend output contracts to include gates + multiplier |
|
||||
| 13 | GATE-3405-013 | DONE | After #3 | Reachability Team | GateDetectionTests.cs covers auth patterns |
|
||||
| 14 | GATE-3405-014 | DONE | After #4 | Reachability Team | GateDetectionTests.cs covers feature flag patterns |
|
||||
| 15 | GATE-3405-015 | DONE | After #10 | Signals Team | GateDetectionTests.cs covers multiplier calculation |
|
||||
| 16 | GATE-3405-016 | BLOCKED | After #11 | QA | Blocked by #11 integration |
|
||||
| 16 | GATE-3405-016 | DONE | After #11 | QA | Add integration coverage for gate propagation + multiplier effect |
|
||||
| 17 | GATE-3405-017 | DONE | After #12 | Docs Guild | Created `docs/reachability/gates.md` |
|
||||
|
||||
## Wave Coordination
|
||||
@@ -585,9 +585,10 @@ public sealed record ReportedGate
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-14 | Sprint created from Determinism advisory gap analysis | Implementer |
|
||||
| 2025-12-18 | Restarted after accidental restore; resuming GATE-3405-009/011/012/016 implementation. | Agent |
|
||||
| 2025-12-18 | Completed Signals gate multiplier scoring + evidence contracts + deterministic integration coverage (GATE-3405-011/012/016). | Agent |
|
||||
| 2025-12-18 | Completed RichGraph gate annotations + JSON writer output; reachability tests green (GATE-3405-009). | Agent |
|
||||
|
||||
## Next Checkpoints
|
||||
|
||||
- Integrate gate detection into RichGraph builder/writer (GATE-3405-009).
|
||||
- Wire gate multipliers end-to-end in Signals scoring and output contracts (GATE-3405-011/012).
|
||||
- Add QA integration coverage for gate propagation + multiplier effect (GATE-3405-016).
|
||||
- None (sprint exit ready). Consider updating downstream report renderers if they need gate visualisation.
|
||||
|
||||
@@ -0,0 +1,721 @@
|
||||
Here are two practical ways to make your software supply‑chain evidence both *useful* and *verifiable*—with enough background to get you shipping.
|
||||
|
||||
---
|
||||
|
||||
# 1) Binary SBOMs that still work when there’s no package manager
|
||||
|
||||
**Why this matters:** Container images built `FROM scratch` or “distroless” often lack package metadata, so typical SBOMs go blank. A *binary SBOM* extracts facts directly from executables—so you still know “what’s inside,” even in bare images.
|
||||
|
||||
**Core idea (plain English):**
|
||||
|
||||
* Parse binaries (ELF on Linux, PE on Windows, Mach‑O on macOS).
|
||||
* Record file paths, cryptographic hashes, import tables, compiler/linker hints, and for ELF also the `.note.gnu.build-id` (a unique ID most linkers embed).
|
||||
* Map these fingerprints to known packages/versions (vendor fingerprints, distro databases, your own allowlists).
|
||||
* Sign the result as an attestation so others can trust it without re‑running your scanner.
|
||||
|
||||
**Minimal pipeline sketch:**
|
||||
|
||||
* **Extract:** `readelf -n` (ELF notes), `objdump`/`otool` for imports; compute SHA‑256 for every binary.
|
||||
* **Normalize:** Emit CycloneDX or SPDX components for *binaries*, not just packages.
|
||||
* **Map:** Use Build‑ID → package hints (e.g., glibc, OpenSSL), symbol/version patterns, and path heuristics.
|
||||
* **Attest:** Wrap the SBOM in DSSE + in‑toto and push to your registry alongside the image digest.
|
||||
|
||||
**Pragmatic spec for developers:**
|
||||
|
||||
* Inputs: OCI image digest.
|
||||
* Outputs:
|
||||
|
||||
* `binary-sbom.cdx.json` (CycloneDX) or `binary-sbom.spdx.json`.
|
||||
* `attestation.intoto.jsonl` (DSSE envelope referencing the SBOM’s SHA‑256 and the *image digest*).
|
||||
* Data fields to capture per artifact:
|
||||
|
||||
* `algorithm: sha256`, `digest: <hex>`, `type: elf|pe|macho`, `path`, `size`,
|
||||
* `elf.build_id` (if present), `imports[]`, `compiler[]`, `arch`, `endian`.
|
||||
* Verification:
|
||||
|
||||
* `cosign verify-attestation --type sbom --digest <image-digest> ...`
|
||||
|
||||
**Why the ELF Build‑ID is gold:** it’s a stable, linker‑emitted identifier that helps correlate stripped binaries to upstream packages—critical when filenames and symbols lie.
|
||||
|
||||
---
|
||||
|
||||
# 2) Reachability analysis so you only page people for *real* risk
|
||||
|
||||
**Why this matters:** Not every CVE in your deps can actually be hit by your app. If you can show “no call path reaches the vulnerable sink,” you can *de‑noise* alerts and ship faster.
|
||||
|
||||
**Core idea (plain English):**
|
||||
|
||||
* Build an *interprocedural call graph* of your app (across modules/packages).
|
||||
* Mark known “sinks” from vulnerability advisories (e.g., dangerous API + version range).
|
||||
* Compute graph reachability from your entrypoints (HTTP handlers, CLI `main`, background jobs).
|
||||
* The intersection of {reachable nodes} × {vulnerable sinks} = “actionable” findings.
|
||||
* Emit a signed *witness* (attestation) that states which sinks are reachable/unreachable and why.
|
||||
|
||||
**Minimal pipeline sketch:**
|
||||
|
||||
* **Ingest code/bytecode:** language‑specific frontends (e.g., .NET IL, JVM bytecode, Python AST, Go SSA).
|
||||
* **Build graph:** nodes = functions/methods; edges = call sites (include dynamic edges conservatively).
|
||||
* **Mark entrypoints:** web routes, message handlers, cron jobs, exported CLIs.
|
||||
* **Mark sinks:** from your vuln DB (API signature + version).
|
||||
* **Decide:** run graph search from entrypoints → is any sink reachable?
|
||||
* **Attest:** DSSE witness with:
|
||||
|
||||
* artifact digest (commit SHA / image digest),
|
||||
* tool version + rule set hash,
|
||||
* list of reachable sinks with at least one example call path,
|
||||
* list of *proven* unreachable sinks (under stated assumptions).
|
||||
|
||||
**Developer contract (portable across languages):**
|
||||
|
||||
* Inputs: source/bytecode zip + manifest of entrypoints.
|
||||
* Outputs:
|
||||
|
||||
* `reachability.witness.json` (DSSE envelope),
|
||||
* optional `paths/` folder with top‑N call paths as compact JSON (for UX rendering).
|
||||
* Verification:
|
||||
|
||||
* Recompute call graph deterministically given the same inputs + tool version,
|
||||
* `cosign verify-attestation --type reachability ...`
|
||||
|
||||
---
|
||||
|
||||
# How these two pieces fit together
|
||||
|
||||
* **Binary SBOM** = “What exactly is in the artifact?” (even in bare images)
|
||||
* **Reachability witness** = “Which vulns actually matter to *this* app build?”
|
||||
* Sign both as **DSSE/in‑toto attestations** and attach to the image/release. Your CI can enforce:
|
||||
|
||||
* “Block if high‑severity + *reachable*,”
|
||||
* “Warn (don’t block) if high‑severity but *unreachable* with a fresh witness.”
|
||||
|
||||
---
|
||||
|
||||
# Quick starter checklist (copy/paste to a task board)
|
||||
|
||||
* [ ] Binary extractors: ELF/PE/Mach‑O parsers; hash & Build‑ID capture.
|
||||
* [ ] Mapping rules: Build‑ID → known package DB; symbol/version heuristics.
|
||||
* [ ] Emit CycloneDX/SPDX; add file‑level components for binaries.
|
||||
* [ ] DSSE signing and `cosign`/`rekor` publish for SBOM attestation.
|
||||
* [ ] Language frontends for reachability (pick your top 1–2 first).
|
||||
* [ ] Call‑graph builder + entrypoint detector.
|
||||
* [ ] Sink catalog normalizer (map CVE → API signature).
|
||||
* [ ] Reachability engine + example path extractor.
|
||||
* [ ] DSSE witness for reachability; attach to build.
|
||||
* [ ] CI policy: block on “reachable high/critical”; surface paths in UI.
|
||||
|
||||
If you want, I can turn this into concrete .NET‑first tasks with sample code scaffolds and a tiny demo repo that builds an image, extracts a binary SBOM, runs reachability on a toy service, and emits both attestations.
|
||||
Below is a concrete, “do‑this‑then‑this” implementation plan for a **layered binary→PURL mapping system** that fits StellaOps’ constraints: **offline**, **deterministic**, **SBOM‑first**, and with **unknowns recorded instead of guessing**.
|
||||
|
||||
I’m going to assume your target is the common pain case StellaOps itself calls out: when package metadata is missing, Scanner falls back to binary identity (`bin:{sha256}`) and you want to deterministically “lift” those binaries into stable package identities (PURLs) without turning the core SBOM into fuzzy guesswork. StellaOps’ own Scanner docs emphasize **deterministic analyzers**, **no fuzzy identity in core**, and keeping heuristics as opt‑in add‑ons. ([Stella Ops][1])
|
||||
|
||||
---
|
||||
|
||||
## 0) What “binary mapping” means in StellaOps terms
|
||||
|
||||
In Scanner’s architecture, the **component key** is:
|
||||
|
||||
* **PURL when present**
|
||||
* otherwise `bin:{sha256}` ([Stella Ops][1])
|
||||
|
||||
So “better binary mapping” = systematically converting more of those `bin:*` components into **PURLs** (or at least producing **actionable mapping evidence + Unknowns**) while preserving:
|
||||
|
||||
* deterministic replay (same inputs ⇒ same output)
|
||||
* offline operation (air‑gapped kits)
|
||||
* policy safety (don’t hide false negatives behind fuzzy IDs)
|
||||
|
||||
Also, StellaOps already has the concept of “gaps” being first‑class via the **Unknowns Registry** (identity gaps, missing build‑id, version conflicts, missing edges, etc.). ([Gitea: Git with a cup of tea][2]) Your binary mapping work should *feed* this system.
|
||||
|
||||
---
|
||||
|
||||
## 1) Design constraints you must keep (or you’ll fight the platform)
|
||||
|
||||
### 1.1 Determinism rules
|
||||
|
||||
StellaOps’ Scanner architecture is explicit: core analyzers are deterministic; heuristic plug‑ins must not contaminate the core SBOM unless explicitly enabled. ([Stella Ops][1])
|
||||
|
||||
That implies:
|
||||
|
||||
* **No probabilistic “best guess” PURL** in the default mapping path.
|
||||
* If you do fuzzy inference, it must be emitted as:
|
||||
|
||||
* “hints” attached to Unknowns, or
|
||||
* a separate heuristic artifact gated by flags.
|
||||
|
||||
### 1.2 Offline kit + debug store is already a hook you can exploit
|
||||
|
||||
Offline kits already bundle:
|
||||
|
||||
* scanner plug‑ins (OS + language analyzers packaged under `plugins/scanner/analyzers/**`)
|
||||
* a **debug store** layout: `debug/.build-id/<aa>/<rest>.debug`
|
||||
* a `debug-manifest.json` that maps build‑ids → originating images (for symbol retrieval) ([Stella Ops][3])
|
||||
|
||||
This is perfect for building a **Build‑ID→PURL index** that remains offline and signed.
|
||||
|
||||
### 1.3 Scanner Worker already loads analyzers via directory catalogs
|
||||
|
||||
The Worker loads OS and language analyzer plug‑ins from default directories (unless overridden), using deterministic directory normalization and a “seal” concept on the last directory. ([Gitea: Git with a cup of tea][4])
|
||||
|
||||
So you can add a third catalog for **native/binary mapping** that behaves the same way.
|
||||
|
||||
---
|
||||
|
||||
## 2) Layering strategy: what to implement (and in what order)
|
||||
|
||||
You want a **resolver pipeline** with strict ordering from “hard evidence” → “soft evidence”.
|
||||
|
||||
### Layer 0 — In‑image authoritative mapping (highest confidence)
|
||||
|
||||
These sources are authoritative because they come from within the artifact:
|
||||
|
||||
1. **OS package DB present** (dpkg/rpm/apk):
|
||||
|
||||
* Map `path → package` using file ownership lists.
|
||||
* If you can also compute file hashes/build‑ids, store them as evidence.
|
||||
|
||||
2. **Language ecosystem metadata present** (already handled by language analyzers):
|
||||
|
||||
* For example, a Python wheel RECORD or a Go buildinfo section can directly imply module versions.
|
||||
|
||||
**Decision rule**: If a binary file is owned by an OS package, **prefer that** over any external mapping index.
|
||||
|
||||
### Layer 1 — “Build provenance” mapping via build IDs / UUIDs (strong, portable)
|
||||
|
||||
When package DB is missing (distroless/scratch), use **compiler/linker stable IDs**:
|
||||
|
||||
* ELF: `.note.gnu.build-id`
|
||||
* Mach‑O: `LC_UUID`
|
||||
* PE: CodeView (PDB GUID+Age) / build signature
|
||||
|
||||
This should be your primary fallback because it survives stripping and renaming.
|
||||
|
||||
### Layer 2 — Hash mapping for curated or vendor‑pinned binaries (strong but brittle across rebuilds)
|
||||
|
||||
Use SHA‑256 → PURL mapping when:
|
||||
|
||||
* binaries are redistributed unchanged (busybox, chromium, embedded runtimes)
|
||||
* you maintain a curated “known binaries” manifest
|
||||
|
||||
StellaOps already has “curated binary manifest generation” mentioned in its repo history, and a `vendor/manifest.json` concept exists (for pinned artifacts / binaries in the system). ([Gitea: Git with a cup of tea][5])
|
||||
For your ops environment you’ll create a similar manifest **for your fleet**.
|
||||
|
||||
### Layer 3 — Dependency closure constraints (helpful as a disambiguator, not a primary mapper)
|
||||
|
||||
If the binary’s DT_NEEDED / imports point to libs you *can* identify, you can use that to disambiguate multiple possible candidates (“this openssl build-id matches, but only one candidate has the required glibc baseline”).
|
||||
|
||||
This must remain deterministic and rules‑based.
|
||||
|
||||
### Layer 4 — Heuristic hints (never change the core SBOM by default)
|
||||
|
||||
Examples:
|
||||
|
||||
* symbol version patterns (`GLIBC_2.28`, etc.)
|
||||
* embedded version strings
|
||||
* import tables
|
||||
* compiler metadata
|
||||
|
||||
These produce **Unknown evidence/hints**, not a resolved identity, unless a special “heuristics allowed” flag is turned on.
|
||||
|
||||
### Layer 5 — Unknowns Registry output (mandatory when you can’t decide)
|
||||
|
||||
If a mapping can’t be made decisively:
|
||||
|
||||
* emit Unknowns (identity_gap, missing_build_id, version_conflict, etc.) ([Gitea: Git with a cup of tea][2])
|
||||
This is not optional; it’s how you prevent silent false negatives.
|
||||
|
||||
---
|
||||
|
||||
## 3) Concrete data model you should implement
|
||||
|
||||
### 3.1 Binary identity record
|
||||
|
||||
Create a single canonical identity structure that *every layer* uses:
|
||||
|
||||
```csharp
|
||||
public enum BinaryFormat { Elf, Pe, MachO, Unknown }
|
||||
|
||||
public sealed record BinaryIdentity(
|
||||
BinaryFormat Format,
|
||||
string Path, // normalized (posix style), rooted at image root
|
||||
string Sha256, // always present
|
||||
string? BuildId, // ELF
|
||||
string? MachOUuid, // Mach-O
|
||||
string? PeCodeViewGuid, // PE/PDB
|
||||
string? Arch, // amd64/arm64/...
|
||||
long SizeBytes
|
||||
);
|
||||
```
|
||||
|
||||
**Determinism tip**: normalize `Path` to a single separator and collapse `//`, `./`, etc.
|
||||
|
||||
### 3.2 Mapping candidate
|
||||
|
||||
Each resolver layer returns candidates like:
|
||||
|
||||
```csharp
|
||||
public enum MappingVerdict { Resolved, Unresolved, Ambiguous }
|
||||
|
||||
public sealed record BinaryMappingCandidate(
|
||||
string Purl,
|
||||
double Confidence, // 0..1 but deterministic
|
||||
string ResolverId, // e.g. "os.fileowner", "buildid.index.v1"
|
||||
IReadOnlyList<string> Evidence, // stable ordering
|
||||
IReadOnlyDictionary<string,string> Properties // stable ordering
|
||||
);
|
||||
```
|
||||
|
||||
### 3.3 Final mapping result
|
||||
|
||||
```csharp
|
||||
public sealed record BinaryMappingResult(
|
||||
MappingVerdict Verdict,
|
||||
BinaryIdentity Subject,
|
||||
BinaryMappingCandidate? Winner,
|
||||
IReadOnlyList<BinaryMappingCandidate> Alternatives,
|
||||
string MappingIndexDigest // sha256 of index snapshot used (or "none")
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4) Build the “Binary Map Index” that makes Layer 1 and 2 work offline
|
||||
|
||||
### 4.1 Where it lives in StellaOps
|
||||
|
||||
Put it in the Offline Kit as a signed artifact, next to other feeds and plug-ins. Offline kit packaging already includes plug-ins and a debug store with a deterministic layout. ([Stella Ops][3])
|
||||
|
||||
Recommended layout:
|
||||
|
||||
```
|
||||
offline-kit/
|
||||
feeds/
|
||||
binary-map/
|
||||
v1/
|
||||
buildid.map.zst
|
||||
sha256.map.zst
|
||||
index.manifest.json
|
||||
index.manifest.json.sig (DSSE or JWS, consistent with your kit)
|
||||
```
|
||||
|
||||
### 4.2 Index record schema (v1)
|
||||
|
||||
Make each record explicit and replayable:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema": "stellaops.binary-map.v1",
|
||||
"records": [
|
||||
{
|
||||
"key": { "kind": "elf.build_id", "value": "2f3a..."},
|
||||
"purl": "pkg:deb/debian/openssl@3.0.11-1~deb12u2?arch=amd64",
|
||||
"evidence": {
|
||||
"source": "os.dpkg.fileowner",
|
||||
"source_image": "sha256:....",
|
||||
"path": "/usr/lib/x86_64-linux-gnu/libssl.so.3",
|
||||
"package": "openssl",
|
||||
"package_version": "3.0.11-1~deb12u2"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
* `key.kind` is one of `elf.build_id`, `macho.uuid`, `pe.codeview`, `file.sha256`
|
||||
* include evidence with enough detail to justify mapping
|
||||
|
||||
### 4.3 How to *generate* the index (deterministically)
|
||||
|
||||
You need an **offline index builder** pipeline. In StellaOps terms, this is best treated like a feed exporter step (build-time), then shipped in the Offline Kit.
|
||||
|
||||
**Input set options** (choose one or mix):
|
||||
|
||||
1. “Golden base images” list (your fleet’s base images)
|
||||
2. Distro repositories mirrored into the airgap (Deb/RPM/APK archives)
|
||||
3. Previously scanned images that are allowed into the kit
|
||||
|
||||
**Generation steps**:
|
||||
|
||||
1. For each input image:
|
||||
|
||||
* Extract rootfs in a deterministic path order.
|
||||
* Run OS analyzers (dpkg/rpm/apk) + native identity collection (ELF/PE/MachO).
|
||||
2. Produce raw tuples:
|
||||
|
||||
* `(build_id | uuid | codeview | sha256) → (purl, evidence)`
|
||||
3. Deduplicate:
|
||||
|
||||
* Canonicalize PURLs (normalize qualifiers order, lowercasing rules).
|
||||
* If the same key maps to **multiple distinct PURLs**, keep them all and mark as conflict (do not pick one).
|
||||
4. Sort:
|
||||
|
||||
* Sort by `(key.kind, key.value, purl)` lexicographically.
|
||||
5. Serialize:
|
||||
|
||||
* Emit line‑delimited JSON or a simple binary format.
|
||||
* Compress (zstd).
|
||||
6. Compute digests:
|
||||
|
||||
* `sha256` of each artifact.
|
||||
* `sha256` of concatenated `(artifact name + sha)` for a manifest hash.
|
||||
7. Sign:
|
||||
|
||||
* include in kit manifest and sign with the same process you use for other offline kit elements. Offline kit import in StellaOps validates digests and signatures. ([Stella Ops][3])
|
||||
|
||||
---
|
||||
|
||||
## 5) Runtime side: implement the layered resolver in Scanner Worker
|
||||
|
||||
### 5.1 Where to hook in
|
||||
|
||||
You want this to run after OS + language analyzers have produced fragments, and after native identity collection has produced binary identities.
|
||||
|
||||
Scanner Worker already executes analyzers and appends fragments to `context.Analysis`. ([Gitea: Git with a cup of tea][4])
|
||||
|
||||
Scanner module responsibilities explicitly include OS, language, and native ecosystems as restart-only plug-ins. ([Gitea: Git with a cup of tea][6])
|
||||
So implement binary mapping as either:
|
||||
|
||||
* part of the **native ecosystem analyzer output stage**, or
|
||||
* a **post-analyzer enrichment stage** that runs before SBOM composition.
|
||||
|
||||
I recommend: **post-analyzer enrichment stage**, because it can consult OS+lang analyzer results and unify decisions.
|
||||
|
||||
### 5.2 Add a new ScanAnalysis key
|
||||
|
||||
Store collected binary identities in analysis:
|
||||
|
||||
* `ScanAnalysisKeys.NativeBinaryIdentities` → `ImmutableArray<BinaryIdentity>`
|
||||
|
||||
And store mapping results:
|
||||
|
||||
* `ScanAnalysisKeys.NativeBinaryMappings` → `ImmutableArray<BinaryMappingResult>`
|
||||
|
||||
### 5.3 Implement the resolver pipeline (deterministic ordering)
|
||||
|
||||
```csharp
|
||||
public interface IBinaryMappingResolver
|
||||
{
|
||||
string Id { get; } // stable ID
|
||||
int Order { get; } // deterministic
|
||||
BinaryMappingCandidate? TryResolve(BinaryIdentity identity, MappingContext ctx);
|
||||
}
|
||||
```
|
||||
|
||||
Pipeline:
|
||||
|
||||
1. Sort resolvers by `(Order, Id)` (Ordinal comparison).
|
||||
2. For each resolver:
|
||||
|
||||
* if it returns a candidate, add it to candidates list.
|
||||
* if the resolver is “authoritative” (Layer 0), you can short‑circuit on first hit.
|
||||
3. Decide:
|
||||
|
||||
* If 0 candidates ⇒ `Unresolved`
|
||||
* If 1 candidate ⇒ `Resolved`
|
||||
* If >1:
|
||||
|
||||
* If candidates have different PURLs ⇒ `Ambiguous` unless a deterministic “dominates” rule exists
|
||||
* If candidates have same PURL (from multiple sources) ⇒ merge evidence
|
||||
|
||||
### 5.4 Implement each layer as a resolver
|
||||
|
||||
#### Resolver A: OS file owner (Layer 0)
|
||||
|
||||
Inputs:
|
||||
|
||||
* OS analyzer results in `context.Analysis` (they’re already stored in `ScanAnalysisKeys.OsPackageAnalyzers`). ([Gitea: Git with a cup of tea][4])
|
||||
* You need OS analyzers to expose file ownership mapping.
|
||||
|
||||
Implementation options:
|
||||
|
||||
* Extend OS analyzers to produce `path → packageId` maps.
|
||||
* Or load that from dpkg/rpm DB at mapping time (fast enough if you only query per binary path).
|
||||
|
||||
Candidate:
|
||||
|
||||
* `Purl = pkg:<ecosystem>/<name>@<version>?arch=...`
|
||||
* Confidence = `1.0`
|
||||
* Evidence includes:
|
||||
|
||||
* analyzer id
|
||||
* package name/version
|
||||
* file path
|
||||
|
||||
#### Resolver B: Build‑ID index (Layer 1)
|
||||
|
||||
Inputs:
|
||||
|
||||
* `identity.BuildId` (or uuid/codeview)
|
||||
* `BinaryMapIndex` loaded from Offline Kit `feeds/binary-map/v1/buildid.map.zst`
|
||||
|
||||
Implementation:
|
||||
|
||||
* On worker startup: load and parse index into an immutable structure:
|
||||
|
||||
* `FrozenDictionary<string, BuildIdEntry[]>` (or sorted arrays + binary search)
|
||||
* If key maps to multiple PURLs:
|
||||
|
||||
* return multiple candidates (same resolver id), forcing `Ambiguous` verdict upstream
|
||||
|
||||
Candidate:
|
||||
|
||||
* Confidence = `0.95` (still deterministic)
|
||||
* Evidence includes index manifest digest + record evidence
|
||||
|
||||
#### Resolver C: SHA‑256 index (Layer 2)
|
||||
|
||||
Inputs:
|
||||
|
||||
* `identity.Sha256`
|
||||
* `feeds/binary-map/v1/sha256.map.zst` OR your ops “curated binaries” manifest
|
||||
|
||||
Candidate:
|
||||
|
||||
* Confidence:
|
||||
|
||||
* `0.9` if from signed curated manifest
|
||||
* `0.7` if from “observed in previous scan cache” (I’d avoid this unless you version and sign the cache)
|
||||
|
||||
#### Resolver D: Dependency closure constraints (Layer 3)
|
||||
|
||||
Only run if you have native dependency parsing output (DT_NEEDED / imports). The resolver does **not** return a mapping on its own; instead, it can:
|
||||
|
||||
* bump confidence for existing candidates
|
||||
* or rule out candidates deterministically (e.g., glibc baseline mismatch)
|
||||
|
||||
Make this a “candidate rewriter” stage:
|
||||
|
||||
```csharp
|
||||
public interface ICandidateRefiner
|
||||
{
|
||||
string Id { get; }
|
||||
int Order { get; }
|
||||
IReadOnlyList<BinaryMappingCandidate> Refine(BinaryIdentity id, IReadOnlyList<BinaryMappingCandidate> cands, MappingContext ctx);
|
||||
}
|
||||
```
|
||||
|
||||
#### Resolver E: Heuristic hints (Layer 4)
|
||||
|
||||
Never resolves to a PURL by default. It just produces Unknown evidence payload:
|
||||
|
||||
* extracted strings (“OpenSSL 3.0.11”)
|
||||
* imported symbol names
|
||||
* SONAME
|
||||
* symbol version requirements
|
||||
|
||||
---
|
||||
|
||||
## 6) SBOM composition behavior: how to “lift” bin components safely
|
||||
|
||||
### 6.1 Don’t break the component key rules
|
||||
|
||||
Scanner uses:
|
||||
|
||||
* key = PURL when present, else `bin:{sha256}` ([Stella Ops][1])
|
||||
|
||||
When you resolve a binary identity to a PURL, you have two clean options:
|
||||
|
||||
**Option 1 (recommended): replace the component key with the PURL**
|
||||
|
||||
* This makes downstream policy/advisory matching work naturally.
|
||||
* It’s deterministic as long as the mapping index is versioned and shipped with the kit.
|
||||
|
||||
**Option 2: keep `bin:{sha256}` as the component key and attach `resolved_purl`**
|
||||
|
||||
* Lower disruption to diffing, but policy now has to understand the “resolved_purl” field.
|
||||
* If StellaOps policy assumes `component.purl` is the canonical key, this will cause pain.
|
||||
|
||||
Given StellaOps emphasizes PURLs as the canonical key for identity, I’d implement **Option 1**, but record robust evidence + index digest.
|
||||
|
||||
### 6.2 Preserve file-level evidence
|
||||
|
||||
Even after lifting to PURL, keep evidence that ties the package identity back to file bytes:
|
||||
|
||||
* file path(s)
|
||||
* sha256
|
||||
* build-id/uuid
|
||||
* mapping resolver id + index digest
|
||||
|
||||
This is what makes attestations verifiable and helps operators debug.
|
||||
|
||||
---
|
||||
|
||||
## 7) Unknowns integration: emit Unknowns whenever mapping isn’t decisive
|
||||
|
||||
The Unknowns Registry exists precisely for “unresolved symbol → package mapping”, “missing build-id”, “ambiguous purl”, etc. ([Gitea: Git with a cup of tea][2])
|
||||
|
||||
### 7.1 When to emit Unknowns
|
||||
|
||||
Emit Unknowns for:
|
||||
|
||||
1. `identity.BuildId == null` for ELF
|
||||
|
||||
* `unknown_type = missing_build_id`
|
||||
* evidence: “ELF missing .note.gnu.build-id; using sha256 only”
|
||||
|
||||
2. Multiple candidates with different PURLs
|
||||
|
||||
* `unknown_type = version_conflict` (or `identity_gap`)
|
||||
* evidence: list candidates + their evidence
|
||||
|
||||
3. Heuristic hints found but no authoritative mapping
|
||||
|
||||
* `unknown_type = identity_gap`
|
||||
* evidence: imported symbols, strings, SONAME
|
||||
|
||||
### 7.2 How to compute `unknown_id` deterministically
|
||||
|
||||
Unknowns schema suggests:
|
||||
|
||||
* `unknown_id` is derived from sha256 over `(type + scope + evidence)` ([Gitea: Git with a cup of tea][2])
|
||||
|
||||
Do:
|
||||
|
||||
* stable JSON canonicalization of `scope` + `unknown_type` + `primary evidence fields`
|
||||
* sha256
|
||||
* prefix with `unk:sha256:<...>`
|
||||
|
||||
This guarantees idempotent ingestion behavior (`POST /unknowns/ingest` upsert). ([Gitea: Git with a cup of tea][2])
|
||||
|
||||
---
|
||||
|
||||
## 8) Packaging as a StellaOps plug-in (so ops can upgrade it offline)
|
||||
|
||||
### 8.1 Plug-in manifest
|
||||
|
||||
Scanner plug-ins use a `manifest.json` with `schemaVersion`, `id`, `entryPoint` (dotnet assembly + typeName), etc. ([Gitea: Git with a cup of tea][7])
|
||||
|
||||
Create something like:
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": "1.0",
|
||||
"id": "stellaops.analyzer.native.binarymap",
|
||||
"displayName": "StellaOps Native Binary Mapper",
|
||||
"version": "0.1.0",
|
||||
"requiresRestart": true,
|
||||
"entryPoint": {
|
||||
"type": "dotnet",
|
||||
"assembly": "StellaOps.Scanner.Analyzers.Native.BinaryMap.dll",
|
||||
"typeName": "StellaOps.Scanner.Analyzers.Native.BinaryMap.BinaryMapPlugin"
|
||||
},
|
||||
"capabilities": [
|
||||
"native-analyzer",
|
||||
"binary-mapper",
|
||||
"elf",
|
||||
"pe",
|
||||
"macho"
|
||||
],
|
||||
"metadata": {
|
||||
"org.stellaops.analyzer.kind": "native",
|
||||
"org.stellaops.restart.required": "true"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 8.2 Worker loading
|
||||
|
||||
Mirror the pattern in `CompositeScanAnalyzerDispatcher`:
|
||||
|
||||
* add a catalog `INativeAnalyzerPluginCatalog`
|
||||
* default directory: `plugins/scanner/analyzers/native`
|
||||
* load directories with the same “seal last directory” behavior ([Gitea: Git with a cup of tea][4])
|
||||
|
||||
---
|
||||
|
||||
## 9) Tests and performance gates (what “done” looks like)
|
||||
|
||||
StellaOps has determinism tests and golden fixtures for analyzers; follow that style. ([Gitea: Git with a cup of tea][6])
|
||||
|
||||
### 9.1 Determinism tests
|
||||
|
||||
Create fixtures with:
|
||||
|
||||
* same binaries in different file order
|
||||
* same binaries hardlinked/symlinked
|
||||
* stripped ELF missing build-id
|
||||
* multi-arch variants
|
||||
|
||||
Assert:
|
||||
|
||||
* mapping output JSON byte-for-byte stable
|
||||
* unknown ids stable
|
||||
* candidate ordering stable
|
||||
|
||||
### 9.2 “No fuzzy identity” guardrail tests
|
||||
|
||||
Add tests that:
|
||||
|
||||
* heuristic resolver never emits a `Resolved` verdict unless a feature flag is enabled
|
||||
* ambiguous candidates never auto-select a winner
|
||||
|
||||
### 9.3 Performance budgets
|
||||
|
||||
For ops, you care about scan wall time. Adopt budgets like:
|
||||
|
||||
* identity extraction < 25ms / binary (native parsing)
|
||||
* mapping lookup O(1) / binary (frozen dict) or O(log n) with sorted arrays
|
||||
* index load time bounded (lazy load per worker start)
|
||||
|
||||
Track metrics:
|
||||
|
||||
* count resolved per layer
|
||||
* count ambiguous/unresolved
|
||||
* unknown density (ties into Unknowns Registry scoring later) ([Gitea: Git with a cup of tea][2])
|
||||
|
||||
---
|
||||
|
||||
## 10) Practical “ops” workflow: how to keep improving mapping safely
|
||||
|
||||
### 10.1 Add a feedback loop from Unknowns → index builder
|
||||
|
||||
Unknowns are your backlog:
|
||||
|
||||
* “missing build-id”
|
||||
* “ambiguous mapping”
|
||||
* “hash seen but not in index”
|
||||
|
||||
For each Unknown:
|
||||
|
||||
1. decide if it should be mapped in core (needs authoritative source)
|
||||
2. if yes: add reference artifact to your **index builder input set**
|
||||
3. rebuild the BinaryMap index
|
||||
4. ship via Offline Kit update (signed)
|
||||
|
||||
### 10.2 Don’t let your index silently drift
|
||||
|
||||
Because determinism matters, treat the BinaryMap index like a feed:
|
||||
|
||||
* version it (`v1`, `v2`)
|
||||
* sign it
|
||||
* store index digest in scan evidence
|
||||
|
||||
That way you can explain: “This binary was mapped using binary-map/v1 digest XYZ”.
|
||||
|
||||
---
|
||||
|
||||
## 11) Minimal implementation checklist (if you want the shortest path to value)
|
||||
|
||||
If you only do 3 things, do these:
|
||||
|
||||
1. **Build‑ID extraction everywhere** (ELF/Mach‑O/PE) and always store it in evidence
|
||||
(also emit Unknown when missing, as StellaOps expects) ([Gitea: Git with a cup of tea][8])
|
||||
|
||||
2. **Offline Build‑ID → PURL index** shipped in Offline Kit
|
||||
(fits perfectly with the existing debug-store + kit pattern) ([Stella Ops][3])
|
||||
|
||||
3. **Deterministic resolver pipeline + Unknowns emission**
|
||||
(so you improve mapping without introducing silent risk) ([Gitea: Git with a cup of tea][2])
|
||||
|
||||
---
|
||||
|
||||
If you tell me whether your main pain is **distroless**, **FROM scratch**, or **vendor‑bundled runtimes** (chromium/node/openssl/etc.), I can give you the best “Layer 1 index builder” recipe for that category (what to use as authoritative sources and how to avoid collisions) — but the plan above is already safe and implementable without further assumptions.
|
||||
|
||||
[1]: https://stella-ops.org/docs/modules/scanner/architecture/ "Stella Ops – Signed Reachability · Deterministic Replay · Sovereign Crypto"
|
||||
[2]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/d519782a8f0b30f425c9b6ae0f316b19259972a2/docs/signals/unknowns-registry.md "git.stella-ops.org/unknowns-registry.md at d519782a8f0b30f425c9b6ae0f316b19259972a2 - git.stella-ops.org - Gitea: Git with a cup of tea"
|
||||
[3]: https://stella-ops.org/docs/24_offline_kit/index.html "Stella Ops – Signed Reachability · Deterministic Replay · Sovereign Crypto"
|
||||
[4]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/18f28168f022c73736bfd29033c71daef5e11044/src/Scanner/StellaOps.Scanner.Worker/Processing/CompositeScanAnalyzerDispatcher.cs "git.stella-ops.org/CompositeScanAnalyzerDispatcher.cs at 18f28168f022c73736bfd29033c71daef5e11044 - git.stella-ops.org - Gitea: Git with a cup of tea"
|
||||
[5]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/8d78dd219b5e44c835e511491a4750f4a3ee3640/vendor/manifest.json?utm_source=chatgpt.com "git.stella-ops.org/manifest.json at ..."
|
||||
[6]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/bc0762e97d251723854b9c4e482b218c8efb1e04/docs/modules/scanner "git.stella-ops.org/scanner at bc0762e97d251723854b9c4e482b218c8efb1e04 - git.stella-ops.org - Gitea: Git with a cup of tea"
|
||||
[7]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/c37722993137dac4b3a4104045826ca33b9dc289/plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/manifest.json "git.stella-ops.org/manifest.json at c37722993137dac4b3a4104045826ca33b9dc289 - git.stella-ops.org - Gitea: Git with a cup of tea"
|
||||
[8]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/d519782a8f0b30f425c9b6ae0f316b19259972a2/docs/reachability/evidence-schema.md?utm_source=chatgpt.com "git.stella-ops.org/evidence-schema.md at ..."
|
||||
@@ -0,0 +1,919 @@
|
||||
Here’s a compact, practical way to add two high‑leverage capabilities to your scanner: **DSSE‑signed path witnesses** and **Smart‑Diff × Reachability**—what they are, why they matter, and exactly how to implement them in Stella Ops without ceremony.
|
||||
|
||||
---
|
||||
|
||||
# 1) DSSE‑signed path witnesses (entrypoint → calls → sink)
|
||||
|
||||
**What it is (in plain terms):**
|
||||
When you flag a CVE as “reachable,” also emit a tiny, human‑readable proof: the **exact path** from a real entrypoint (e.g., HTTP route, CLI verb, cron) through functions/methods to the **vulnerable sink**. Wrap that proof in a **DSSE** envelope and sign it. Anyone can verify the witness later—offline—without rerunning analysis.
|
||||
|
||||
**Why it matters:**
|
||||
|
||||
* Turns red flags into **auditable evidence** (quiet‑by‑design).
|
||||
* Lets CI/CD, auditors, and customers **verify** findings independently.
|
||||
* Enables **deterministic replay** and provenance chains (ties nicely to in‑toto/SLSA).
|
||||
|
||||
**Minimal JSON witness (stable, vendor‑neutral):**
|
||||
|
||||
```json
|
||||
{
|
||||
"witness_schema": "stellaops.witness.v1",
|
||||
"artifact": { "sbom_digest": "sha256:...", "component_purl": "pkg:nuget/Example@1.2.3" },
|
||||
"vuln": { "id": "CVE-2024-XXXX", "source": "NVD", "range": "≤1.2.3" },
|
||||
"entrypoint": { "kind": "http", "name": "GET /billing/pay" },
|
||||
"path": [
|
||||
{"symbol": "BillingController.Pay()", "file": "BillingController.cs", "line": 42},
|
||||
{"symbol": "PaymentsService.Authorize()", "file": "PaymentsService.cs", "line": 88},
|
||||
{"symbol": "LibXYZ.Parser.Parse()", "file": "Parser.cs", "line": 17}
|
||||
],
|
||||
"sink": { "symbol": "LibXYZ.Parser.Parse()", "type": "deserialization" },
|
||||
"evidence": {
|
||||
"callgraph_digest": "sha256:...",
|
||||
"build_id": "dotnet:RID:linux-x64:sha256:...",
|
||||
"analysis_config_digest": "sha256:..."
|
||||
},
|
||||
"observed_at": "2025-12-18T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**Wrap in DSSE (payloadType & payload are required)**
|
||||
|
||||
```json
|
||||
{
|
||||
"payloadType": "application/vnd.stellaops.witness+json",
|
||||
"payload": "base64(JSON_above)",
|
||||
"signatures": [{ "keyid": "attestor-stellaops-ed25519", "sig": "base64(...)" }]
|
||||
}
|
||||
```
|
||||
|
||||
**.NET 10 signing/verifying (Ed25519)**
|
||||
|
||||
```csharp
|
||||
using System.Security.Cryptography;
|
||||
using System.Text.Json;
|
||||
|
||||
var payloadBytes = JsonSerializer.SerializeToUtf8Bytes(witnessJsonObj);
|
||||
var dsse = new {
|
||||
payloadType = "application/vnd.stellaops.witness+json",
|
||||
payload = Convert.ToBase64String(payloadBytes),
|
||||
signatures = new [] { new { keyid = keyId, sig = Convert.ToBase64String(Sign(payloadBytes, privateKey)) } }
|
||||
};
|
||||
byte[] Sign(byte[] data, byte[] privateKey)
|
||||
{
|
||||
using var ed = new Ed25519();
|
||||
// import private key, sign data (left as your Ed25519 helper)
|
||||
return ed.SignData(data, privateKey);
|
||||
}
|
||||
```
|
||||
|
||||
**Where to emit:**
|
||||
|
||||
* **Scanner.Worker**: after reachability confirms `reachable=true`, emit witness → **Attestor** signs → **Authority** stores (Postgres) → optional Rekor‑style mirror.
|
||||
* Expose `/witness/{findingId}` for download & independent verification.
|
||||
|
||||
---
|
||||
|
||||
# 2) Smart‑Diff × Reachability (incremental, low‑noise updates)
|
||||
|
||||
**What it is:**
|
||||
On **SBOM/VEX/dependency** deltas, don’t rescan everything. Update only **affected regions** of the call graph and recompute reachability **just for changed nodes/edges**.
|
||||
|
||||
**Why it matters:**
|
||||
|
||||
* **Order‑of‑magnitude faster** incremental scans.
|
||||
* Fewer flaky diffs; triage stays focused on **meaningful risk change**.
|
||||
* Perfect for PR gating: “what changed” → “what became reachable/unreachable.”
|
||||
|
||||
**Core idea (graph‑reachability):**
|
||||
|
||||
* Maintain a per‑service **call graph** `G = (V, E)` with **entrypoint set** `S`.
|
||||
* On diff: compute changed nodes/edges ΔV/ΔE.
|
||||
* Run **incremental BFS/DFS** from impacted nodes to sinks (forward or backward), reusing memoized results.
|
||||
* Recompute only **frontiers** touched by Δ.
|
||||
|
||||
**Minimal tables (Postgres):**
|
||||
|
||||
```sql
|
||||
-- Nodes (functions/methods)
|
||||
CREATE TABLE cg_nodes(
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
service TEXT, symbol TEXT, file TEXT, line INT,
|
||||
hash TEXT, UNIQUE(service, hash)
|
||||
);
|
||||
-- Edges (calls)
|
||||
CREATE TABLE cg_edges(
|
||||
src BIGINT REFERENCES cg_nodes(id),
|
||||
dst BIGINT REFERENCES cg_nodes(id),
|
||||
kind TEXT, PRIMARY KEY(src, dst)
|
||||
);
|
||||
-- Entrypoints & Sinks
|
||||
CREATE TABLE cg_entrypoints(node_id BIGINT REFERENCES cg_nodes(id) PRIMARY KEY);
|
||||
CREATE TABLE cg_sinks(node_id BIGINT REFERENCES cg_nodes(id) PRIMARY KEY, sink_type TEXT);
|
||||
|
||||
-- Memoized reachability cache
|
||||
CREATE TABLE cg_reach_cache(
|
||||
entry_id BIGINT, sink_id BIGINT,
|
||||
path JSONB, reachable BOOLEAN,
|
||||
updated_at TIMESTAMPTZ,
|
||||
PRIMARY KEY(entry_id, sink_id)
|
||||
);
|
||||
```
|
||||
|
||||
**Incremental algorithm (pseudocode):**
|
||||
|
||||
```text
|
||||
Input: ΔSBOM, ΔDeps, ΔCode → ΔNodes, ΔEdges
|
||||
1) Apply Δ to cg_nodes/cg_edges
|
||||
2) ImpactSet = neighbors(ΔNodes ∪ endpoints(ΔEdges))
|
||||
3) For each e∈Entrypoints intersect ancestors(ImpactSet):
|
||||
Recompute forward search to affected sinks, stop early on unchanged subgraphs
|
||||
Update cg_reach_cache; if state flips, emit new/updated DSSE witness
|
||||
```
|
||||
|
||||
**.NET 10 reachability sketch (fast & local):**
|
||||
|
||||
```csharp
|
||||
HashSet<int> ImpactSet = ComputeImpact(deltaNodes, deltaEdges);
|
||||
foreach (var e in Intersect(Entrypoints, Ancestors(ImpactSet)))
|
||||
{
|
||||
var res = BoundedReach(e, affectedSinks, graph, cache);
|
||||
foreach (var r in res.Changed)
|
||||
{
|
||||
cache.Upsert(e, r.Sink, r.Path, r.Reachable);
|
||||
if (r.Reachable) EmitDsseWitness(e, r.Sink, r.Path);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**CI/PR flow:**
|
||||
|
||||
1. Build → SBOM diff → Dependency diff → Call‑graph delta.
|
||||
2. Run incremental reachability.
|
||||
3. If any `unreachable→reachable` transitions: **fail gate**, attach DSSE witnesses.
|
||||
4. If `reachable→unreachable`: auto‑close prior findings (and archive prior witness).
|
||||
|
||||
---
|
||||
|
||||
# UX hooks (quick wins)
|
||||
|
||||
* In findings list, add a **“Show Witness”** button → modal renders the signed path (entrypoint→…→sink) + **“Verify Signature”** one‑click.
|
||||
* In PR checks, summarize only **state flips** with tiny links: “+2 reachable (view witness)” / “−1 (now unreachable)”.
|
||||
|
||||
---
|
||||
|
||||
# Minimal tasks to get this live
|
||||
|
||||
* **Scanner.Worker**: build call‑graph extraction (per language), add incremental graph store, reachability cache.
|
||||
* **Attestor**: DSSE signing endpoint + key management (Ed25519 by default; PQC mode later).
|
||||
* **Authority**: tables above + witness storage + retrieval API.
|
||||
* **Router/CI plugin**: PR annotation with **state flips** and links to witnesses.
|
||||
* **UI**: witness modal + signature verify.
|
||||
|
||||
If you want, I can draft the exact Postgres migrations, the C# repositories, and a tiny verifier CLI that checks DSSE signatures and prints the call path.
|
||||
Below is a concrete, buildable blueprint for an **advanced reachability analysis engine** inside Stella Ops. I’m going to assume your “Stella Ops” components are roughly:
|
||||
|
||||
* **Scanner.Worker**: runs analyses in CI / on artifacts
|
||||
* **Authority**: stores graphs/findings/witnesses
|
||||
* **Attestor**: signs DSSE envelopes (Ed25519)
|
||||
* (optional) **SurfaceBuilder**: background worker that computes “vuln surfaces” for packages
|
||||
|
||||
The key advance is: **don’t treat a CVE as “a package”**. Treat it as a **set of trigger methods** (public API) that can reach the vulnerable code inside the dependency—computed by “Smart‑Diff” once, reused everywhere.
|
||||
|
||||
---
|
||||
|
||||
## 0) Define the contract (precision/soundness) up front
|
||||
|
||||
If you don’t write this down, you’ll fight false positives/negatives forever.
|
||||
|
||||
### What Stella Ops will guarantee (first release)
|
||||
|
||||
* **Whole-program static call graph** (app + selected dependency assemblies)
|
||||
* **Context-insensitive** (fast), **path witness** extracted (shortest path)
|
||||
* **Dynamic dispatch handled** with CHA/RTA (+ DI hints), with explicit uncertainty flags
|
||||
* **Reflection handled best-effort** (constant-string resolution), otherwise “unknown edge”
|
||||
|
||||
### What it will NOT guarantee (first release)
|
||||
|
||||
* Perfect handling of reflection / `dynamic` / runtime codegen
|
||||
* Perfect delegate/event resolution across complex flows
|
||||
* Full taint/dataflow reachability (you can add later)
|
||||
|
||||
This is fine. The major value is: “**we can show you the call path**” and “**we can prove the vuln is triggered by calling these library APIs**”.
|
||||
|
||||
---
|
||||
|
||||
## 1) The big idea: “Vuln surfaces” (Smart-Diff → triggers)
|
||||
|
||||
### Problem
|
||||
|
||||
CVE feeds typically say “package X version range Y is vulnerable” but rarely say *which methods*. If you only do package-level reachability, noise is huge.
|
||||
|
||||
### Solution
|
||||
|
||||
For each CVE+package, compute a **vulnerability surface**:
|
||||
|
||||
* **Candidate sinks** = methods changed between vulnerable and fixed versions (diff at IL level)
|
||||
* **Trigger methods** = *public/exported* methods in the vulnerable version that can reach those changed methods internally
|
||||
|
||||
Then your service scan becomes:
|
||||
|
||||
> “Can any entrypoint reach any trigger method?”
|
||||
|
||||
This is both faster and more precise.
|
||||
|
||||
---
|
||||
|
||||
## 2) Data model (Authority / Postgres)
|
||||
|
||||
You already had call graph tables; here’s a concrete schema that supports:
|
||||
|
||||
* graph snapshots
|
||||
* incremental updates
|
||||
* vuln surfaces
|
||||
* reachability cache
|
||||
* DSSE witnesses
|
||||
|
||||
### 2.1 Graph tables
|
||||
|
||||
```sql
|
||||
CREATE TABLE cg_snapshots (
|
||||
snapshot_id BIGSERIAL PRIMARY KEY,
|
||||
service TEXT NOT NULL,
|
||||
build_id TEXT NOT NULL,
|
||||
graph_digest TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
UNIQUE(service, build_id)
|
||||
);
|
||||
|
||||
CREATE TABLE cg_nodes (
|
||||
node_id BIGSERIAL PRIMARY KEY,
|
||||
snapshot_id BIGINT REFERENCES cg_snapshots(snapshot_id) ON DELETE CASCADE,
|
||||
method_key TEXT NOT NULL, -- stable key (see below)
|
||||
asm_name TEXT,
|
||||
type_name TEXT,
|
||||
method_name TEXT,
|
||||
file_path TEXT,
|
||||
line_start INT,
|
||||
il_hash TEXT, -- normalized IL hash for diffing
|
||||
flags INT NOT NULL DEFAULT 0, -- bitflags: has_reflection, compiler_generated, etc.
|
||||
UNIQUE(snapshot_id, method_key)
|
||||
);
|
||||
|
||||
CREATE TABLE cg_edges (
|
||||
snapshot_id BIGINT REFERENCES cg_snapshots(snapshot_id) ON DELETE CASCADE,
|
||||
src_node_id BIGINT REFERENCES cg_nodes(node_id) ON DELETE CASCADE,
|
||||
dst_node_id BIGINT REFERENCES cg_nodes(node_id) ON DELETE CASCADE,
|
||||
kind SMALLINT NOT NULL, -- 0=call,1=newobj,2=dispatch,3=delegate,4=reflection_guess,...
|
||||
PRIMARY KEY(snapshot_id, src_node_id, dst_node_id, kind)
|
||||
);
|
||||
|
||||
CREATE TABLE cg_entrypoints (
|
||||
snapshot_id BIGINT REFERENCES cg_snapshots(snapshot_id) ON DELETE CASCADE,
|
||||
node_id BIGINT REFERENCES cg_nodes(node_id) ON DELETE CASCADE,
|
||||
kind TEXT NOT NULL, -- http, grpc, cli, job, etc.
|
||||
name TEXT NOT NULL, -- GET /foo, "Main", etc.
|
||||
PRIMARY KEY(snapshot_id, node_id, kind, name)
|
||||
);
|
||||
```
|
||||
|
||||
### 2.2 Vuln surface tables (Smart‑Diff artifacts)
|
||||
|
||||
```sql
|
||||
CREATE TABLE vuln_surfaces (
|
||||
surface_id BIGSERIAL PRIMARY KEY,
|
||||
ecosystem TEXT NOT NULL, -- nuget
|
||||
package TEXT NOT NULL,
|
||||
cve_id TEXT NOT NULL,
|
||||
vuln_version TEXT NOT NULL, -- a representative vulnerable version
|
||||
fixed_version TEXT NOT NULL,
|
||||
surface_digest TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
UNIQUE(ecosystem, package, cve_id, vuln_version, fixed_version)
|
||||
);
|
||||
|
||||
CREATE TABLE vuln_surface_sinks (
|
||||
surface_id BIGINT REFERENCES vuln_surfaces(surface_id) ON DELETE CASCADE,
|
||||
sink_method_key TEXT NOT NULL,
|
||||
reason TEXT NOT NULL, -- changed|added|removed|heuristic
|
||||
PRIMARY KEY(surface_id, sink_method_key)
|
||||
);
|
||||
|
||||
CREATE TABLE vuln_surface_triggers (
|
||||
surface_id BIGINT REFERENCES vuln_surfaces(surface_id) ON DELETE CASCADE,
|
||||
trigger_method_key TEXT NOT NULL,
|
||||
sink_method_key TEXT NOT NULL,
|
||||
internal_path JSONB, -- optional: library internal witness path
|
||||
PRIMARY KEY(surface_id, trigger_method_key, sink_method_key)
|
||||
);
|
||||
```
|
||||
|
||||
### 2.3 Reachability cache & witnesses
|
||||
|
||||
```sql
|
||||
CREATE TABLE reach_findings (
|
||||
finding_id BIGSERIAL PRIMARY KEY,
|
||||
snapshot_id BIGINT REFERENCES cg_snapshots(snapshot_id) ON DELETE CASCADE,
|
||||
cve_id TEXT NOT NULL,
|
||||
ecosystem TEXT NOT NULL,
|
||||
package TEXT NOT NULL,
|
||||
package_version TEXT NOT NULL,
|
||||
reachable BOOLEAN NOT NULL,
|
||||
reachable_entrypoints INT NOT NULL DEFAULT 0,
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
UNIQUE(snapshot_id, cve_id, package, package_version)
|
||||
);
|
||||
|
||||
CREATE TABLE reach_witnesses (
|
||||
witness_id BIGSERIAL PRIMARY KEY,
|
||||
finding_id BIGINT REFERENCES reach_findings(finding_id) ON DELETE CASCADE,
|
||||
entry_node_id BIGINT REFERENCES cg_nodes(node_id),
|
||||
dsse_envelope JSONB NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3) Stable identity: MethodKey + IL hash
|
||||
|
||||
### 3.1 MethodKey (must be stable across builds)
|
||||
|
||||
Use a normalized string like:
|
||||
|
||||
```
|
||||
{AssemblyName}|{DeclaringTypeFullName}|{MethodName}`{GenericArity}({ParamType1},{ParamType2},...)
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
* `MyApp|BillingController|Pay(System.String)`
|
||||
* `LibXYZ|LibXYZ.Parser|Parse(System.ReadOnlySpan<System.Byte>)`
|
||||
|
||||
### 3.2 Normalized IL hash (for smart-diff + incremental graph updates)
|
||||
|
||||
Raw IL bytes aren’t stable (metadata tokens change). Normalize:
|
||||
|
||||
* opcode names
|
||||
* branch targets by *instruction index*, not offset
|
||||
* method operands by **resolved MethodKey**
|
||||
* string operands by literal or hashed literal
|
||||
* type operands by full name
|
||||
|
||||
Then hash `SHA256(normalized_bytes)`.
|
||||
|
||||
---
|
||||
|
||||
## 4) Call graph extraction for .NET (concrete, doable)
|
||||
|
||||
### Tooling choice
|
||||
|
||||
Start with **Mono.Cecil** (MIT license, easy IL traversal). You can later swap to `System.Reflection.Metadata` for speed.
|
||||
|
||||
### 4.1 Build process (Scanner.Worker)
|
||||
|
||||
1. `dotnet restore` (use your locked restore)
|
||||
2. `dotnet build -c Release /p:DebugType=portable /p:DebugSymbols=true`
|
||||
3. Collect:
|
||||
|
||||
* app assemblies: `bin/Release/**/publish/*.dll` or build output
|
||||
* `.pdb` files for sequence points (file/line for witnesses)
|
||||
|
||||
### 4.2 Cecil loader
|
||||
|
||||
```csharp
|
||||
var rp = new ReaderParameters {
|
||||
ReadSymbols = true,
|
||||
SymbolReaderProvider = new PortablePdbReaderProvider()
|
||||
};
|
||||
|
||||
var asm = AssemblyDefinition.ReadAssembly(dllPath, rp);
|
||||
```
|
||||
|
||||
### 4.3 Node extraction (methods)
|
||||
|
||||
Walk all types, including nested:
|
||||
|
||||
```csharp
|
||||
IEnumerable<TypeDefinition> AllTypes(ModuleDefinition m)
|
||||
{
|
||||
var stack = new Stack<TypeDefinition>(m.Types);
|
||||
while (stack.Count > 0)
|
||||
{
|
||||
var t = stack.Pop();
|
||||
yield return t;
|
||||
foreach (var nt in t.NestedTypes) stack.Push(nt);
|
||||
}
|
||||
}
|
||||
|
||||
foreach (var type in AllTypes(asm.MainModule))
|
||||
foreach (var method in type.Methods)
|
||||
{
|
||||
var key = MethodKey.From(method); // your normalizer
|
||||
var (file, line) = PdbFirstSequencePoint(method);
|
||||
var ilHash = method.HasBody ? ILFingerprint(method) : null;
|
||||
|
||||
// store node (method_key, file, line, il_hash, flags...)
|
||||
}
|
||||
```
|
||||
|
||||
### 4.4 Edge extraction (direct calls)
|
||||
|
||||
```csharp
|
||||
foreach (var method in type.Methods.Where(m => m.HasBody))
|
||||
{
|
||||
var srcKey = MethodKey.From(method);
|
||||
foreach (var ins in method.Body.Instructions)
|
||||
{
|
||||
if (ins.Operand is MethodReference mr)
|
||||
{
|
||||
if (ins.OpCode.Code is Code.Call or Code.Callvirt or Code.Newobj)
|
||||
{
|
||||
var dstKey = MethodKey.From(mr); // important: stable even if not resolved
|
||||
edges.Add(new Edge(srcKey, dstKey, kind: CallKind.Direct));
|
||||
}
|
||||
if (ins.OpCode.Code is Code.Ldftn or Code.Ldvirtftn)
|
||||
{
|
||||
// delegate capture (handle later)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5) Advanced precision: dynamic dispatch + DI + async/await
|
||||
|
||||
If you stop at direct edges only, you’ll miss many real paths.
|
||||
|
||||
### 5.1 Async/await mapping (critical for readable witnesses)
|
||||
|
||||
Async methods compile into a state machine `MoveNext()`. You want edges attributed back to the original method.
|
||||
|
||||
In Cecil:
|
||||
|
||||
* Check `AsyncStateMachineAttribute` on a method
|
||||
* It references a state machine type
|
||||
* Find that type’s `MoveNext` method
|
||||
* Map `MoveNextKey -> OriginalMethodKey`
|
||||
|
||||
Then, while extracting edges:
|
||||
|
||||
```csharp
|
||||
srcKey = MoveNextToOriginal.TryGetValue(srcKey, out var original) ? original : srcKey;
|
||||
```
|
||||
|
||||
Do the same for iterator state machines.
|
||||
|
||||
### 5.2 Virtual/interface dispatch (CHA/RTA)
|
||||
|
||||
You need 2 maps:
|
||||
|
||||
1. **type hierarchy / interface impl map**
|
||||
2. **override map** from “declared method” → “implementation method(s)”
|
||||
|
||||
**Build override map**
|
||||
|
||||
```csharp
|
||||
// For each method, Cecil exposes method.Overrides for explicit implementations.
|
||||
overrideMap[MethodKey.From(overrideRef)] = MethodKey.From(methodDef);
|
||||
```
|
||||
|
||||
**CHA**: for callvirt to virtual method `T.M`, add edges to overrides in derived classes
|
||||
**RTA**: restrict to derived classes that are actually instantiated.
|
||||
|
||||
How to get instantiated types:
|
||||
|
||||
* look for `newobj` instructions and add the created type to `InstantiatedTypes`
|
||||
* plus DI registrations (below)
|
||||
|
||||
### 5.3 DI hints (Microsoft.Extensions.DependencyInjection)
|
||||
|
||||
You will see calls like:
|
||||
|
||||
* `ServiceCollectionServiceExtensions.AddTransient<TService, TImpl>(...)`
|
||||
|
||||
In IL these are generic method calls. Detect and record `TService -> TImpl` as “instantiated”. This massively improves RTA for modern .NET apps.
|
||||
|
||||
### 5.4 Delegates/lambdas (good enough approach)
|
||||
|
||||
Implement intraprocedural tracking:
|
||||
|
||||
* when you see `ldftn SomeMethod` then `newobj Action::.ctor` then `stloc.s X`
|
||||
* store `delegateTargets[local X] += SomeMethod`
|
||||
* when you see `ldloc.s X` and later `callvirt Invoke`, add edges to targets
|
||||
|
||||
This makes Minimal API entrypoint discovery work too.
|
||||
|
||||
### 5.5 Reflection (best-effort)
|
||||
|
||||
Implement only high-signal heuristics:
|
||||
|
||||
* `typeof(T).GetMethod("Foo")` with constant "Foo"
|
||||
* `GetType().GetMethod("Foo")` with constant "Foo" (type unknown → mark uncertain)
|
||||
|
||||
If resolved, add edge with `kind=reflection_guess`.
|
||||
If not, set node flag `has_reflection = true` and in results show “may be incomplete”.
|
||||
|
||||
---
|
||||
|
||||
## 6) Entrypoint detection (concrete detectors)
|
||||
|
||||
### 6.1 MVC controllers
|
||||
|
||||
Detect:
|
||||
|
||||
* types deriving from `Microsoft.AspNetCore.Mvc.ControllerBase`
|
||||
* methods:
|
||||
|
||||
* public
|
||||
* not `[NonAction]`
|
||||
* has `[HttpGet]`, `[HttpPost]`, `[Route]` etc.
|
||||
|
||||
Extract route template from attributes’ ctor arguments.
|
||||
|
||||
Store in `cg_entrypoints`:
|
||||
|
||||
* kind = `http`
|
||||
* name = `GET /billing/pay` (compose verb+template)
|
||||
|
||||
### 6.2 Minimal APIs
|
||||
|
||||
Scan `Program.Main` IL:
|
||||
|
||||
* find calls to `MapGet`, `MapPost`, ...
|
||||
* extract route string from preceding `ldstr`
|
||||
* resolve handler method via delegate tracking (ldftn)
|
||||
|
||||
Entry:
|
||||
|
||||
* kind = `http`
|
||||
* name = `GET /foo`
|
||||
|
||||
### 6.3 CLI
|
||||
|
||||
Find assembly entry point method (`asm.EntryPoint`) or `static Main`.
|
||||
Entry:
|
||||
|
||||
* kind = `cli`
|
||||
* name = `Main`
|
||||
|
||||
Start here. Add gRPC/jobs later.
|
||||
|
||||
---
|
||||
|
||||
## 7) Smart-Diff SurfaceBuilder (the “advanced” part)
|
||||
|
||||
This is what makes your reachability actually meaningful for CVEs.
|
||||
|
||||
### 7.1 SurfaceBuilder inputs
|
||||
|
||||
From your vuln ingestion pipeline:
|
||||
|
||||
* ecosystem = nuget
|
||||
* package = `LibXYZ`
|
||||
* affected range = `<= 1.2.3`
|
||||
* fixed version = `1.2.4`
|
||||
* CVE id
|
||||
|
||||
### 7.2 Choose a vulnerable version to diff
|
||||
|
||||
Pick the **highest affected version below fixed**.
|
||||
|
||||
* fixed = 1.2.4
|
||||
* vulnerable representative = 1.2.3
|
||||
|
||||
(If multiple fixed versions exist, build multiple surfaces.)
|
||||
|
||||
### 7.3 Download both packages
|
||||
|
||||
Use NuGet.Protocol to download `.nupkg`, unzip, pick TFMs you care about (often `netstandard2.0` is safest). Compute fingerprints for each assembly.
|
||||
|
||||
### 7.4 Compute method fingerprints
|
||||
|
||||
For each method:
|
||||
|
||||
* MethodKey
|
||||
* Normalized IL hash
|
||||
|
||||
### 7.5 Diff
|
||||
|
||||
```
|
||||
ChangedMethods = { k | hashVuln[k] != hashFixed[k] } ∪ added ∪ removed
|
||||
```
|
||||
|
||||
Store these as `vuln_surface_sinks` with reason.
|
||||
|
||||
### 7.6 Build internal library call graph
|
||||
|
||||
Same Cecil extraction, but only for package assemblies.
|
||||
Now compute triggers:
|
||||
|
||||
**Reverse BFS from sinks**:
|
||||
|
||||
* Start from all sink method keys
|
||||
* Walk predecessors
|
||||
* When you encounter a **public/exported method**, record it as a trigger
|
||||
|
||||
Also store one internal path for each trigger → sink (for witnesses).
|
||||
|
||||
### 7.7 Add interface/base declarations as triggers
|
||||
|
||||
Important: your app might call a library via an interface method signature, not the concrete implementation.
|
||||
|
||||
For each trigger implementation method:
|
||||
|
||||
* for each `method.Overrides` entry, add the overridden method key as an additional trigger
|
||||
|
||||
This reduces dependence on perfect dispatch expansion during app scanning.
|
||||
|
||||
### 7.8 Persist the surface
|
||||
|
||||
Store:
|
||||
|
||||
* sinks set
|
||||
* triggers set
|
||||
* internal witness paths (optional but highly valuable)
|
||||
|
||||
Now you’ve converted a “version range” CVE into “these specific library APIs are dangerous”.
|
||||
|
||||
---
|
||||
|
||||
## 8) Reachability engine (fast, witness-producing)
|
||||
|
||||
### 8.1 In-memory graph format (CSR)
|
||||
|
||||
Don’t BFS off dictionaries; you’ll die on perf.
|
||||
|
||||
Build integer indices:
|
||||
|
||||
* `method_key -> nodeIndex (0..N-1)`
|
||||
* store arrays:
|
||||
|
||||
* `predOffsets[N+1]`
|
||||
* `preds[edgeCount]`
|
||||
|
||||
Construction:
|
||||
|
||||
1. count predecessors per node
|
||||
2. prefix sum to offsets
|
||||
3. fill preds
|
||||
|
||||
### 8.2 Reverse BFS from sinks
|
||||
|
||||
This computes:
|
||||
|
||||
* `visited[node]` = can reach a sink
|
||||
* `parent[node]` = next node toward a sink (for path reconstruction)
|
||||
|
||||
```csharp
|
||||
public sealed class ReachabilityEngine
|
||||
{
|
||||
public ReachabilityResult Compute(
|
||||
Graph g,
|
||||
ReadOnlySpan<int> entrypoints,
|
||||
ReadOnlySpan<int> sinks)
|
||||
{
|
||||
var visitedMark = g.VisitMark; // int[] length N (reused across runs)
|
||||
var parent = g.Parent; // int[] length N (reused)
|
||||
g.RunId++;
|
||||
|
||||
var q = new IntQueue(capacity: g.NodeCount);
|
||||
var sinkSet = new BitSet(g.NodeCount);
|
||||
foreach (var s in sinks)
|
||||
{
|
||||
sinkSet.Set(s);
|
||||
visitedMark[s] = g.RunId;
|
||||
parent[s] = s;
|
||||
q.Enqueue(s);
|
||||
}
|
||||
|
||||
while (q.TryDequeue(out var v))
|
||||
{
|
||||
var start = g.PredOffsets[v];
|
||||
var end = g.PredOffsets[v + 1];
|
||||
for (int i = start; i < end; i++)
|
||||
{
|
||||
var p = g.Preds[i];
|
||||
if (visitedMark[p] == g.RunId) continue;
|
||||
visitedMark[p] = g.RunId;
|
||||
parent[p] = v;
|
||||
q.Enqueue(p);
|
||||
}
|
||||
}
|
||||
|
||||
// Collect reachable entrypoints and paths
|
||||
var results = new List<EntryWitness>();
|
||||
foreach (var e in entrypoints)
|
||||
{
|
||||
if (visitedMark[e] != g.RunId) continue;
|
||||
var path = ReconstructPath(e, parent, sinkSet);
|
||||
results.Add(new EntryWitness(e, path));
|
||||
}
|
||||
|
||||
return new ReachabilityResult(results);
|
||||
}
|
||||
|
||||
private static int[] ReconstructPath(int entry, int[] parent, BitSet sinks)
|
||||
{
|
||||
var path = new List<int>(32);
|
||||
int cur = entry;
|
||||
path.Add(cur);
|
||||
|
||||
// follow parent pointers until a sink
|
||||
for (int guard = 0; guard < 10_000; guard++)
|
||||
{
|
||||
if (sinks.Get(cur)) break;
|
||||
var nxt = parent[cur];
|
||||
if (nxt == cur || nxt < 0) break; // safety
|
||||
cur = nxt;
|
||||
path.Add(cur);
|
||||
}
|
||||
return path.ToArray();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 8.3 Producing the witness
|
||||
|
||||
For each node index in the path:
|
||||
|
||||
* method_key
|
||||
* file_path / line_start (if known)
|
||||
* optional flags (reflection_guess edge, dispatch edge)
|
||||
|
||||
Then attach:
|
||||
|
||||
* vuln id, package, version
|
||||
* entrypoint kind/name
|
||||
* graph digest + config digest
|
||||
* surface digest
|
||||
* timestamp
|
||||
|
||||
Send JSON to Attestor for DSSE signing, store envelope in Authority.
|
||||
|
||||
---
|
||||
|
||||
## 9) Scaling: don’t do BFS 500 times if you can avoid it
|
||||
|
||||
### 9.1 First-line scaling (usually enough)
|
||||
|
||||
* Group vulnerabilities by package/version → surfaces reused
|
||||
* Only run reachability for vulns where:
|
||||
|
||||
* dependency present AND
|
||||
* surface exists OR fallback mode
|
||||
* Limit witnesses per vuln (top 3)
|
||||
|
||||
In practice, with N~50k nodes and E~200k edges, a reverse BFS is fast in C# if done with arrays.
|
||||
|
||||
### 9.2 Incremental Smart-Diff × Reachability (your “low noise” killer feature)
|
||||
|
||||
#### Step A: compute graph delta between snapshots
|
||||
|
||||
Use `il_hash` per method to detect changed nodes:
|
||||
|
||||
* added / removed / changed nodes
|
||||
* edges updated only for changed nodes
|
||||
|
||||
#### Step B: decide which vulnerabilities need recompute
|
||||
|
||||
Store a cached reverse-reachable set per vuln surface if you want (bitset), OR just do a cheaper heuristic:
|
||||
|
||||
Recompute for vulnerability if:
|
||||
|
||||
* sink set changed (new surface or version changed), OR
|
||||
* any changed node is on any previously stored witness path, OR
|
||||
* entrypoints changed, OR
|
||||
* impacted nodes touch any trigger node’s predecessors (use a small localized search)
|
||||
|
||||
A practical approach:
|
||||
|
||||
* store all node IDs that appear in any witness path for that vuln
|
||||
* if delta touches any of those nodes/edges, recompute
|
||||
* otherwise reuse cached result
|
||||
|
||||
This yields a massive win on PR scans where most code is unchanged.
|
||||
|
||||
#### Step C: “Impact frontier” recompute (optional)
|
||||
|
||||
If you want more advanced:
|
||||
|
||||
* compute `ImpactSet = ΔNodes ∪ endpoints(ΔEdges)`
|
||||
* run reverse BFS **starting from ImpactSet ∩ ReverseReachSet** and update visited marks
|
||||
This is trickier to implement correctly (dynamic graph), so I’d ship the heuristic first.
|
||||
|
||||
---
|
||||
|
||||
## 10) Practical fallback modes (don’t block shipping)
|
||||
|
||||
You won’t have surfaces for every CVE on day 1. Handle this gracefully:
|
||||
|
||||
### Mode 1: Surface-based reachability (best)
|
||||
|
||||
* sink = trigger methods from surface
|
||||
* result: “reachable” with path
|
||||
|
||||
### Mode 2: Package API usage (good fallback)
|
||||
|
||||
* sink = *any* method in that package that is called by app
|
||||
* result: “package reachable” (lower confidence), still provide path to callsite
|
||||
|
||||
### Mode 3: Dependency present only (SBOM level)
|
||||
|
||||
* no call graph needed
|
||||
* result: “present” only
|
||||
|
||||
Your UI can show confidence tiers:
|
||||
|
||||
* **Confirmed reachable (surface)**
|
||||
* **Likely reachable (package API)**
|
||||
* **Present only (SBOM)**
|
||||
|
||||
---
|
||||
|
||||
## 11) Integration points inside Stella Ops
|
||||
|
||||
### Scanner.Worker (per build)
|
||||
|
||||
1. Build/collect assemblies + pdb
|
||||
2. `CallGraphBuilder` → nodes/edges/entrypoints + graph_digest
|
||||
3. Load SBOM vulnerabilities list
|
||||
4. For each vuln:
|
||||
|
||||
* resolve surface triggers; if missing → enqueue SurfaceBuilder job + fallback mode
|
||||
* run reachability BFS
|
||||
* for each reachable entrypoint: emit DSSE witness
|
||||
5. Persist findings/witnesses
|
||||
|
||||
### SurfaceBuilder (async worker)
|
||||
|
||||
* triggered by “surface missing” events or nightly preload of top packages
|
||||
* computes surface once, stores forever
|
||||
|
||||
### Authority
|
||||
|
||||
* stores graphs, surfaces, findings, witnesses
|
||||
* provides retrieval APIs for UI/CI
|
||||
|
||||
---
|
||||
|
||||
## 12) What to implement first (in the order that produces value fastest)
|
||||
|
||||
### Week 1–2 scope (realistic, shippable)
|
||||
|
||||
1. Cecil call graph extraction (direct calls)
|
||||
2. MVC + Minimal API entrypoints
|
||||
3. Reverse BFS reachability with path witnesses
|
||||
4. DSSE witness signing + storage
|
||||
5. SurfaceBuilder v1:
|
||||
|
||||
* IL hash per method
|
||||
* changed methods as sinks
|
||||
* triggers via internal reverse BFS
|
||||
6. UI: “Show Witness” + “Verify Signature”
|
||||
|
||||
### Next increment (precision upgrades)
|
||||
|
||||
7. async/await mapping to original methods
|
||||
8. RTA + DI registration hints
|
||||
9. delegate tracking for Minimal API handlers (if not already)
|
||||
10. interface override triggers in surface builder
|
||||
|
||||
### Later (if you want “attackability”, not just “reachability”)
|
||||
|
||||
11. taint/dataflow for top sink classes (deserialization, path traversal, SQL, command exec)
|
||||
12. sanitizer modeling & parameter constraints
|
||||
|
||||
---
|
||||
|
||||
## 13) Common failure modes and how to harden
|
||||
|
||||
### MethodKey mismatches (surface vs app call)
|
||||
|
||||
* Ensure both are generated from the same normalization rules
|
||||
* For generic methods, prefer **definition** keys (strip instantiation)
|
||||
* Store both “exact” and “erased generic” variants if needed
|
||||
|
||||
### Multi-target frameworks
|
||||
|
||||
* SurfaceBuilder: compute triggers for each TFM, union them
|
||||
* App scan: choose TFM closest to build RID, but allow fallback to union
|
||||
|
||||
### Huge graphs
|
||||
|
||||
* Drop `System.*` nodes/edges unless:
|
||||
|
||||
* the vuln is in System.* (rare, but handle separately)
|
||||
* Deduplicate nodes by MethodKey across assemblies where safe
|
||||
* Use CSR arrays + pooled queues
|
||||
|
||||
### Reflection heavy projects
|
||||
|
||||
* Mark analysis confidence lower
|
||||
* Include “unknown edges present” in finding metadata
|
||||
* Still produce a witness path up to the reflective callsite
|
||||
|
||||
---
|
||||
|
||||
If you want, I can also paste a **complete Cecil-based CallGraphBuilder class** (nodes+edges+PDB lines), plus the **SurfaceBuilder** that downloads NuGet packages and generates `vuln_surface_triggers` end-to-end.
|
||||
@@ -0,0 +1,751 @@
|
||||
Here’s a practical, first‑time‑friendly blueprint for making your security workflow both **explainable** and **provable**—from triage to approval.
|
||||
|
||||
# Explainable triage UX (what & why)
|
||||
|
||||
Show every risk score with the minimum evidence a responder needs to trust it:
|
||||
|
||||
* **Reachable path:** the concrete call‑chain (or network path) proving the vuln is actually hit.
|
||||
* **Entrypoint boundary:** the external surface (HTTP route, CLI verb, cron, message topic) that leads to that path.
|
||||
* **VEX status:** the exploitability decision (Affected/Not Affected/Under Investigation/Fixed) with rationale.
|
||||
* **Last‑seen timestamp:** when this evidence was last observed/generated.
|
||||
|
||||
## UI pattern (compact, 1‑click expand)
|
||||
|
||||
* **Row (collapsed):** `Score 72 • CVE‑2024‑12345 • service: api-gateway • package: x.y.z`
|
||||
* **Expand panel (evidence):**
|
||||
|
||||
* **Path:** `POST /billing/charge → BillingController.Pay() → StripeClient.Create()`
|
||||
* **Boundary:** `Ingress: /billing/charge (JWT: required, scope: payments:write)`
|
||||
* **VEX:** `Not Affected (runtime guard strips untrusted input before sink)`
|
||||
* **Last seen:** `2025‑12‑18T09:22Z` (scan: sbomer#c1a2, policy run: lattice#9f0d)
|
||||
* **Actions:** “Open proof bundle”, “Re-run check”, “Create exception (time‑boxed)”
|
||||
|
||||
## Data contract (what the panel needs)
|
||||
|
||||
```json
|
||||
{
|
||||
"finding_id": "f-7b3c",
|
||||
"cve": "CVE-2024-12345",
|
||||
"component": {"name": "stripe-sdk", "version": "6.1.2"},
|
||||
"reachable_path": [
|
||||
"HTTP POST /billing/charge",
|
||||
"BillingController.Pay",
|
||||
"StripeClient.Create"
|
||||
],
|
||||
"entrypoint": {"type":"http","route":"/billing/charge","auth":"jwt:payments:write"},
|
||||
"vex": {"status":"not_affected","justification":"runtime_sanitizer_blocks_sink","timestamp":"2025-12-18T09:22:00Z"},
|
||||
"last_seen":"2025-12-18T09:22:00Z",
|
||||
"attestation_refs": ["sha256:…sbom", "sha256:…vex", "sha256:…policy"]
|
||||
}
|
||||
```
|
||||
|
||||
# Evidence‑linked approvals (what & why)
|
||||
|
||||
Make “Approve to ship” contingent on **verifiable proof**, not screenshots:
|
||||
|
||||
* **Chain** must exist and be machine‑verifiable: **SBOM → VEX → policy decision**.
|
||||
* Use **in‑toto/DSSE** attestations or **SLSA provenance** so each link has a signature, subject digest, and predicate.
|
||||
* **Gate** merges/deploys only when the chain validates.
|
||||
|
||||
## Pipeline gate (simple policy)
|
||||
|
||||
* Require:
|
||||
|
||||
1. **SBOM attestation** referencing the exact image digest
|
||||
2. **VEX attestation** covering all listed components (or explicit allow‑gaps)
|
||||
3. **Policy decision attestation** (e.g., “risk ≤ threshold AND all reachable vulns = Not Affected/Fixed”)
|
||||
|
||||
### Minimal decision attestation (DSSE envelope → JSON payload)
|
||||
|
||||
```json
|
||||
{
|
||||
"predicateType": "stella/policy-decision@v1",
|
||||
"subject": [{"name":"registry/org/app","digest":{"sha256":"<image-digest>"}}],
|
||||
"predicate": {
|
||||
"policy": "risk_threshold<=75 && reachable_vulns.all(v => v.vex in ['not_affected','fixed'])",
|
||||
"inputs": {
|
||||
"sbom_ref": "sha256:<sbom>",
|
||||
"vex_ref": "sha256:<vex>"
|
||||
},
|
||||
"result": {"allowed": true, "score": 61, "exemptions":[]},
|
||||
"evidence_refs": ["sha256:<reachability-proof-bundle>"],
|
||||
"run_at": "2025-12-18T09:23:11Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
# How this lands in your product (concrete moves)
|
||||
|
||||
* **Backend:** add `/findings/:id/evidence` (returns the contract above) + `/approvals/:artifact/attestations`.
|
||||
* **Storage:** keep **proof bundles** (graphs, call stacks, logs) as content‑addressed blobs; store DSSE envelopes alongside.
|
||||
* **UI:** one list → expandable rows; chips for VEX status; “Open proof” shows the call graph and boundary in 1 view.
|
||||
* **CLI/API:** `stella verify image:<digest> --require sbom,vex,decision` returns a signed summary; pipelines fail on non‑zero.
|
||||
* **Metrics:**
|
||||
|
||||
* **% changes with complete attestations** (target ≥95%)
|
||||
* **TTFE (time‑to‑first‑evidence)** from alert → panel open (target ≤30s)
|
||||
* **Post‑deploy reversions** due to missing proof (trend to zero)
|
||||
|
||||
# Starter acceptance checklist
|
||||
|
||||
* [ ] Every risk row expands to path, boundary, VEX, last‑seen in <300 ms.
|
||||
* [ ] “Approve” button disabled until SBOM+VEX+Decision attestations validate for the **exact artifact digest**.
|
||||
* [ ] One‑click “Show DSSE chain” renders the three envelopes with subject digests and signers.
|
||||
* [ ] Audit log captures who approved, which digests, and which evidence hashes.
|
||||
|
||||
If you want, I can turn this into ready‑to‑drop **.NET 10** endpoints + a small React panel with mocked data so your team can wire it up fast.
|
||||
Below is a “build‑it” guide for Stella Ops that goes past the concept level: concrete services, schemas, pipelines, signing/storage choices, UI components, and the exact invariants you should enforce so triage is **explainable** and approvals are **provably evidence‑linked**.
|
||||
|
||||
---
|
||||
|
||||
## 1) Start with the invariants (the rules your system must never violate)
|
||||
|
||||
If you implement nothing else, implement these invariants—they’re what make the UX trustworthy and the approvals auditable.
|
||||
|
||||
### Artifact anchoring invariant
|
||||
|
||||
Every finding, every piece of evidence, and every approval must be anchored to an immutable **subject digest** (e.g., container image digest `sha256:…`, binary SHA, or SBOM digest).
|
||||
|
||||
* No “latest tag” approvals.
|
||||
* No “approve commit” without mapping to the built artifact digest.
|
||||
|
||||
### Evidence closure invariant
|
||||
|
||||
A policy decision is only valid if it references **exactly** the evidence it used:
|
||||
|
||||
* `inputs.sbom_ref`
|
||||
* `inputs.vex_ref`
|
||||
* `inputs.reachability_ref` (optional but recommended)
|
||||
* `inputs.scan_ref` (optional)
|
||||
* and any config/IaC refs used for boundary/exposure.
|
||||
|
||||
### Signature chain invariant
|
||||
|
||||
Evidence is only admissible if it is:
|
||||
|
||||
1. structured (machine readable),
|
||||
2. signed (DSSE/in‑toto),
|
||||
3. verifiable (trusted identity/keys),
|
||||
4. retrievable by digest.
|
||||
|
||||
DSSE is specifically designed to authenticate both the message and its type (payload type) and avoid canonicalization pitfalls. ([GitHub][1])
|
||||
|
||||
### Staleness invariant
|
||||
|
||||
Evidence must have:
|
||||
|
||||
* `last_seen` and `expires_at` (or TTL),
|
||||
* a “stale evidence” behavior in policy (deny or degrade score).
|
||||
|
||||
---
|
||||
|
||||
## 2) Choose the canonical formats and where you’ll store “proof”
|
||||
|
||||
### Attestation envelope: DSSE + in‑toto Statement
|
||||
|
||||
Use:
|
||||
|
||||
* **in‑toto Attestation Framework** “Statement” as the payload model (“subject + predicateType + predicate”). ([GitHub][2])
|
||||
* Wrap it in **DSSE** for signing. ([GitHub][1])
|
||||
* If you use Sigstore bundles, the DSSE envelope is expected to carry an in‑toto statement and uses `payloadType` like `application/vnd.in-toto+json`. ([Sigstore][3])
|
||||
|
||||
### SBOM format: CycloneDX or SPDX
|
||||
|
||||
* SPDX is an ISO/IEC standard and has v3.0 and v2.3 lines in the ecosystem. ([spdx.dev][4])
|
||||
* CycloneDX is an ECMA standard (ECMA‑424) and widely used for application security contexts. ([GitHub][5])
|
||||
|
||||
Pick one as **your canonical** (internally), but ingest both.
|
||||
|
||||
### VEX format: OpenVEX (practical) + map to “classic” VEX statuses
|
||||
|
||||
VEX’s value is triage noise reduction: vendors can assert whether a product is affected, fixed, under investigation, or not affected. ([NTIA][6])
|
||||
OpenVEX is a minimal, embeddable implementation of VEX intended for interoperability. ([GitHub][7])
|
||||
|
||||
### Where to store proof: OCI registry referrers
|
||||
|
||||
Use OCI “subject/referrers” so proofs travel with the artifact:
|
||||
|
||||
* OCI 1.1 introduces an explicit `subject` field and referrers graph for signatures/attestations/SBOMs. ([opencontainers.org][8])
|
||||
* ORAS documentation explains linking artifacts via `subject`. ([Oras][9])
|
||||
* Microsoft docs show `oras attach … --artifact-type …` patterns (works across registries that support referrers). ([Microsoft Learn][10])
|
||||
|
||||
---
|
||||
|
||||
## 3) System architecture (services + data flow)
|
||||
|
||||
### Services (minimum set)
|
||||
|
||||
1. **Ingestor**
|
||||
|
||||
* Pulls scanner outputs (SCA/SAST/IaC), SBOM, runtime signals.
|
||||
2. **Evidence Builder**
|
||||
|
||||
* Computes reachability, entrypoints, boundary/auth context, score explanation.
|
||||
3. **Attestation Service**
|
||||
|
||||
* Creates in‑toto statements, wraps DSSE, signs (cosign/KMS), stores to registry.
|
||||
4. **Policy Engine**
|
||||
|
||||
* Evaluates allow/deny + reason codes, emits signed decision attestation.
|
||||
* Use OPA/Rego for maintainable declarative policies. ([openpolicyagent.org][11])
|
||||
5. **Stella Ops API**
|
||||
|
||||
* Serves findings + evidence panels to the UI (fast, cached).
|
||||
6. **UI**
|
||||
|
||||
* Explainable triage panel + chain viewer + approve button.
|
||||
|
||||
### Event flow (artifact‑centric)
|
||||
|
||||
1. Build produces `image@sha256:X`
|
||||
2. Generate SBOM → sign + attach
|
||||
3. Run vuln scan → sign + attach (optional but useful)
|
||||
4. Evidence Builder creates:
|
||||
|
||||
* reachability proof
|
||||
* boundary proof
|
||||
* vex doc (or imports vendor VEX + adds your context)
|
||||
5. Policy engine evaluates → emits “decision attestation”
|
||||
6. UI shows explainable triage + “approve” gating
|
||||
|
||||
---
|
||||
|
||||
## 4) Data model (the exact objects you need)
|
||||
|
||||
### Core IDs you should standardize
|
||||
|
||||
* `subject_digest`: `sha256:<image digest>`
|
||||
* `subject_name`: `registry/org/app`
|
||||
* `finding_key`: `(subject_digest, detector, cve, component_purl, location)` stable hash
|
||||
* `component_purl`: package URL (PURL) canonical component identifier
|
||||
|
||||
### Tables (Postgres suggested)
|
||||
|
||||
**artifacts**
|
||||
|
||||
* `id (uuid)`
|
||||
* `name`
|
||||
* `digest` (unique)
|
||||
* `created_at`
|
||||
|
||||
**findings**
|
||||
|
||||
* `id (uuid)`
|
||||
* `artifact_digest`
|
||||
* `cve`
|
||||
* `component_purl`
|
||||
* `severity`
|
||||
* `raw_score`
|
||||
* `risk_score`
|
||||
* `status` (open/triaged/accepted/fixed)
|
||||
* `first_seen`, `last_seen`
|
||||
|
||||
**evidence**
|
||||
|
||||
* `id (uuid)`
|
||||
* `finding_id`
|
||||
* `kind` (reachable_path | boundary | score_explain | vex | ...)
|
||||
* `payload_json` (jsonb, small)
|
||||
* `blob_ref` (content-addressed URI for big payloads)
|
||||
* `last_seen`
|
||||
* `expires_at`
|
||||
* `confidence` (0–1)
|
||||
* `source_attestation_digest` (nullable)
|
||||
|
||||
**attestations**
|
||||
|
||||
* `id (uuid)`
|
||||
* `artifact_digest`
|
||||
* `predicate_type`
|
||||
* `attestation_digest` (sha256 of DSSE envelope)
|
||||
* `signer_identity` (OIDC subject / cert identity)
|
||||
* `issued_at`
|
||||
* `registry_ref` (where attached)
|
||||
|
||||
**approvals**
|
||||
|
||||
* `id (uuid)`
|
||||
* `artifact_digest`
|
||||
* `decision_attestation_digest`
|
||||
* `approver`
|
||||
* `approved_at`
|
||||
* `expires_at`
|
||||
* `reason`
|
||||
|
||||
---
|
||||
|
||||
## 5) Explainable triage: how to compute the “Path + Boundary + VEX + Last‑seen”
|
||||
|
||||
### 5.1 Reachable path proof (call chain / flow)
|
||||
|
||||
You need a uniform reachability result type:
|
||||
|
||||
* `reachable = true` with an explicit path
|
||||
* `reachable = false` with justification (e.g., symbol absent, dead code)
|
||||
* `reachable = unknown` with reason (insufficient symbols, dynamic dispatch)
|
||||
|
||||
**Implementation strategy**
|
||||
|
||||
1. **Symbol mapping**: map CVE → vulnerable symbols/functions/classes
|
||||
|
||||
* Use one or more:
|
||||
|
||||
* vendor advisory → patched functions
|
||||
* diff mining (commit that fixes CVE) to extract changed symbols
|
||||
* curated mapping in your DB for high volume CVEs
|
||||
2. **Program graph extraction** at build time:
|
||||
|
||||
* Produce a call graph or dependency graph per language.
|
||||
* Store as compact adjacency list (or protobuf) keyed by `subject_digest`.
|
||||
3. **Entrypoint discovery**:
|
||||
|
||||
* HTTP routes (framework metadata)
|
||||
* gRPC service methods
|
||||
* queue/stream consumers
|
||||
* cron/CLI handlers
|
||||
4. **Path search**:
|
||||
|
||||
* BFS/DFS from entrypoints to vulnerable symbols.
|
||||
* Record the shortest path + top‑K alternatives.
|
||||
5. **Proof bundle**:
|
||||
|
||||
* path nodes with stable IDs
|
||||
* file hashes + line ranges (no raw source required)
|
||||
* tool version + config hash
|
||||
* graph digest
|
||||
|
||||
**Reachability evidence JSON (UI‑friendly)**
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "reachable_path",
|
||||
"result": "reachable",
|
||||
"confidence": 0.86,
|
||||
"entrypoints": [
|
||||
{"type":"http","route":"POST /billing/charge","auth":"jwt:payments:write"}
|
||||
],
|
||||
"paths": [{
|
||||
"path_id": "p-1",
|
||||
"steps": [
|
||||
{"node":"BillingController.Pay","file_hash":"sha256:aaa","lines":[41,88]},
|
||||
{"node":"StripeClient.Create","file_hash":"sha256:bbb","lines":[10,52]},
|
||||
{"node":"stripe-sdk.vulnFn","symbol":"stripe-sdk::parseWebhook","evidence":"symbol-match"}
|
||||
]
|
||||
}],
|
||||
"graph": {"digest":"sha256:callgraph...", "format":"stella-callgraph-v1"},
|
||||
"last_seen": "2025-12-18T09:22:00Z",
|
||||
"expires_at": "2025-12-25T09:22:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**UI rule:** never show “reachable” without a concrete, replayable path ID.
|
||||
|
||||
---
|
||||
|
||||
### 5.2 Boundary proof (the “why this is exposed” part)
|
||||
|
||||
Boundary proof answers: “Even if reachable, who can trigger it?”
|
||||
|
||||
**Data sources**
|
||||
|
||||
* Kubernetes ingress/service (exposure)
|
||||
* API gateway routes and auth policies
|
||||
* service mesh auth (mTLS, JWT)
|
||||
* IAM policies (for cloud events)
|
||||
* network policies (deny/allow)
|
||||
|
||||
**Boundary evidence schema**
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "boundary",
|
||||
"surface": {"type":"http","route":"POST /billing/charge"},
|
||||
"exposure": {"internet": true, "ports":[443]},
|
||||
"auth": {
|
||||
"mechanism":"jwt",
|
||||
"required_scopes":["payments:write"],
|
||||
"audience":"billing-api"
|
||||
},
|
||||
"rate_limits": {"enabled": true, "rps": 20},
|
||||
"controls": [
|
||||
{"type":"waf","status":"enabled"},
|
||||
{"type":"input_validation","status":"enabled","location":"BillingController.Pay"}
|
||||
],
|
||||
"last_seen": "2025-12-18T09:22:00Z",
|
||||
"confidence": 0.74
|
||||
}
|
||||
```
|
||||
|
||||
**How to build it**
|
||||
|
||||
* Create a “Surface Extractor” plugin per environment:
|
||||
|
||||
* `k8s-extractor`: reads ingress + service + annotations
|
||||
* `gateway-extractor`: reads API gateway config
|
||||
* `iac-extractor`: parses Terraform/CloudFormation
|
||||
* Normalize into the schema above.
|
||||
|
||||
---
|
||||
|
||||
### 5.3 VEX in Stella: statuses + justifications
|
||||
|
||||
VEX statuses you should support in UI:
|
||||
|
||||
* Not affected
|
||||
* Affected
|
||||
* Fixed
|
||||
* Under investigation ([NTIA][6])
|
||||
|
||||
OpenVEX will carry the machine readable structure. ([GitHub][7])
|
||||
|
||||
**Practical approach**
|
||||
|
||||
* Treat VEX as **the decision record** for exploitability.
|
||||
* Your policy can require VEX coverage for all “reachable” high severity vulns.
|
||||
|
||||
**Rule of thumb**
|
||||
|
||||
* If `reachable=true` AND boundary shows reachable surface + auth weak → VEX defaults to `affected` until mitigations proven.
|
||||
* If `reachable=false` with high confidence and stable proof → VEX may be `not_affected`.
|
||||
|
||||
---
|
||||
|
||||
### 5.4 Explainable risk score (don’t hide the formula)
|
||||
|
||||
Make score explainability first‑class.
|
||||
|
||||
**Recommended implementation**
|
||||
|
||||
* Store risk score as an additive model:
|
||||
|
||||
* `base = CVSS normalized`
|
||||
* `+ reachability_bonus`
|
||||
* `+ exposure_bonus`
|
||||
* `+ privilege_bonus`
|
||||
* `- mitigation_discount`
|
||||
* Emit a `score_explain` evidence object:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "score_explain",
|
||||
"risk_score": 72,
|
||||
"contributions": [
|
||||
{"factor":"cvss","value":41,"reason":"CVSS 9.8"},
|
||||
{"factor":"reachability","value":18,"reason":"reachable path p-1"},
|
||||
{"factor":"exposure","value":10,"reason":"internet-facing route"},
|
||||
{"factor":"auth","value":3,"reason":"scope required lowers impact"}
|
||||
],
|
||||
"last_seen":"2025-12-18T09:22:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**UI rule:** “Score 72” must always be clickable to a stable breakdown.
|
||||
|
||||
---
|
||||
|
||||
## 6) The UI you should build (components + interaction rules)
|
||||
|
||||
### 6.1 Findings list row (collapsed)
|
||||
|
||||
Show only what helps scanning:
|
||||
|
||||
* Score badge
|
||||
* CVE + component
|
||||
* service
|
||||
* reachability chip: Reachable / Not reachable / Unknown
|
||||
* VEX chip
|
||||
* last_seen indicator (green/yellow/red)
|
||||
|
||||
### 6.2 Evidence drawer (expanded)
|
||||
|
||||
Tabs:
|
||||
|
||||
1. **Path**
|
||||
|
||||
* show entrypoint(s)
|
||||
* render call chain (simple list first; graph view optional)
|
||||
2. **Boundary**
|
||||
|
||||
* exposure, auth, controls
|
||||
3. **VEX**
|
||||
|
||||
* status + justification + issuer identity
|
||||
4. **Score**
|
||||
|
||||
* breakdown bar/list
|
||||
5. **Proof**
|
||||
|
||||
* attestation chain viewer (SBOM → VEX → Decision)
|
||||
* “Verify locally” action
|
||||
|
||||
### 6.3 “Open proof bundle” viewer
|
||||
|
||||
Must display:
|
||||
|
||||
* subject digest
|
||||
* signer identity
|
||||
* predicate type
|
||||
* digest of proof bundle
|
||||
* last_seen + tool versions
|
||||
|
||||
**This is where trust is built:** responders can see that the evidence is signed, tied to the artifact, and recent.
|
||||
|
||||
---
|
||||
|
||||
## 7) Proof‑linked evidence: how to generate and attach attestations
|
||||
|
||||
### 7.1 Statement format: in‑toto Attestation Framework
|
||||
|
||||
in‑toto’s model is:
|
||||
|
||||
* **Subjects** (the artifact digests)
|
||||
* **Predicate type** (schema ID)
|
||||
* **Predicate** (your actual data) ([GitHub][2])
|
||||
|
||||
### 7.2 DSSE envelope
|
||||
|
||||
Wrap statements using DSSE so payload type is signed too. ([GitHub][1])
|
||||
|
||||
### 7.3 Attach to OCI image via referrers
|
||||
|
||||
OCI “subject/referrers” makes attestations discoverable from the image digest. ([opencontainers.org][8])
|
||||
ORAS provides the operational model (“attach artifacts to an image”). ([Microsoft Learn][10])
|
||||
|
||||
### 7.4 Practical signing: cosign attest + verify
|
||||
|
||||
Cosign has built‑in in‑toto attestation support and can sign custom predicates. ([Sigstore][12])
|
||||
|
||||
Typical patterns (example only; adapt to your environment):
|
||||
|
||||
```bash
|
||||
# Attach an attestation
|
||||
cosign attest --predicate reachability.json \
|
||||
--type stella/reachability/v1 \
|
||||
<image@sha256:digest>
|
||||
|
||||
# Verify attestation
|
||||
cosign verify-attestation --type stella/reachability/v1 \
|
||||
<image@sha256:digest>
|
||||
```
|
||||
|
||||
(Use keyless OIDC or KMS keys depending on your org.)
|
||||
|
||||
---
|
||||
|
||||
## 8) Define your predicate types (this is the “contract” Stella enforces)
|
||||
|
||||
You’ll want at least these predicate types:
|
||||
|
||||
1. `stella/sbom@v1`
|
||||
|
||||
* embeds CycloneDX/SPDX (or references blob digest)
|
||||
|
||||
2. `stella/vex@v1`
|
||||
|
||||
* embeds OpenVEX document or references it ([GitHub][7])
|
||||
|
||||
3. `stella/reachability@v1`
|
||||
|
||||
* the reachability evidence above
|
||||
* includes `graph.digest`, `paths`, `confidence`, `expires_at`
|
||||
|
||||
4. `stella/boundary@v1`
|
||||
|
||||
* exposure/auth proof and `last_seen`
|
||||
|
||||
5. `stella/policy-decision@v1`
|
||||
|
||||
* the gating result, references all input attestation digests
|
||||
|
||||
6. Optional: `stella/human-approval@v1`
|
||||
|
||||
* “I approve deploy of subject digest X based on decision attestation Y”
|
||||
* keep it time‑boxed
|
||||
|
||||
---
|
||||
|
||||
## 9) The policy gate (how approvals become proof‑linked)
|
||||
|
||||
### 9.1 Use OPA/Rego for the gate
|
||||
|
||||
OPA policies are written in Rego. ([openpolicyagent.org][11])
|
||||
|
||||
**Gate input** should be a single JSON document assembled from verified attestations:
|
||||
|
||||
```json
|
||||
{
|
||||
"subject": {"name":"registry/org/app","digest":"sha256:..."},
|
||||
"sbom": {...},
|
||||
"vex": {...},
|
||||
"reachability": {...},
|
||||
"boundary": {...},
|
||||
"org_policy": {"max_risk": 75, "max_age_hours": 168}
|
||||
}
|
||||
```
|
||||
|
||||
**Example Rego (deny‑by‑default)**
|
||||
|
||||
```rego
|
||||
package stella.gate
|
||||
|
||||
default allow := false
|
||||
|
||||
# deny if evidence is stale
|
||||
stale_evidence {
|
||||
now := time.now_ns()
|
||||
exp := time.parse_rfc3339_ns(input.reachability.expires_at)
|
||||
now > exp
|
||||
}
|
||||
|
||||
# deny if any high severity reachable vuln is not resolved by VEX
|
||||
unresolved_reachable[v] {
|
||||
v := input.reachability.findings[_]
|
||||
v.severity in {"critical","high"}
|
||||
v.reachable == true
|
||||
not input.vex.resolution[v.cve] in {"not_affected","fixed"}
|
||||
}
|
||||
|
||||
allow {
|
||||
input.risk_score <= input.org_policy.max_risk
|
||||
not stale_evidence
|
||||
count(unresolved_reachable) == 0
|
||||
}
|
||||
```
|
||||
|
||||
### 9.2 Emit a signed policy decision attestation
|
||||
|
||||
When OPA returns `allow=true`, emit **another attestation**:
|
||||
|
||||
* predicate includes the policy version/hash and all input refs.
|
||||
* that’s what the UI “Approve” button targets.
|
||||
|
||||
This is the “evidence‑linked approval”: approval references the signed decision, and the decision references the signed evidence.
|
||||
|
||||
---
|
||||
|
||||
## 10) “Approve” button behavior (what Stella Ops should enforce)
|
||||
|
||||
### Disabled until…
|
||||
|
||||
* subject digest known
|
||||
* SBOM attestation found + signature verified
|
||||
* VEX attestation found + signature verified
|
||||
* Decision attestation found + signature verified
|
||||
* Decision’s `inputs` digests match the actual retrieved evidence
|
||||
|
||||
### When clicked…
|
||||
|
||||
1. Stella Ops creates a `stella/human-approval@v1` statement:
|
||||
|
||||
* `subject` = artifact digest
|
||||
* `predicate.decision_ref` = decision attestation digest
|
||||
* `predicate.expires_at` = short TTL (e.g., 7–30 days)
|
||||
2. Signs it with the approver identity
|
||||
3. Attaches it to the artifact (OCI referrer)
|
||||
|
||||
### Audit view must show
|
||||
|
||||
* approver identity
|
||||
* exact artifact digest
|
||||
* exact decision attestation digest
|
||||
* timestamp and expiry
|
||||
|
||||
---
|
||||
|
||||
## 11) Implementation details that matter in production
|
||||
|
||||
### 11.1 Verification library (shared by UI backend + CI gate)
|
||||
|
||||
Write one verifier module used everywhere:
|
||||
|
||||
**Inputs**
|
||||
|
||||
* image digest
|
||||
* expected predicate types
|
||||
* trust policy (allowed identities/issuers, keyless rules, KMS keys)
|
||||
|
||||
**Steps**
|
||||
|
||||
1. Discover referrers for `image@sha256:…`
|
||||
2. Filter by `predicateType`
|
||||
3. Verify DSSE + signature + identity
|
||||
4. Validate JSON schema for predicate
|
||||
5. Check `subject.digest` matches image digest
|
||||
6. Return “verified evidence set” + “errors”
|
||||
|
||||
### 11.2 Evidence privacy
|
||||
|
||||
Reachability proofs can leak implementation details.
|
||||
|
||||
* Store file hashes, symbol names, and line ranges
|
||||
* Gate raw source behind elevated permissions
|
||||
* Provide redacted proofs by default
|
||||
|
||||
### 11.3 Evidence TTL strategy
|
||||
|
||||
* SBOM: long TTL (weeks/months) if digest immutable
|
||||
* Boundary: short TTL (hours/days) because env changes
|
||||
* Reachability: medium TTL (days/weeks) depending on code churn
|
||||
* VEX: must be renewed if boundary/reachability changes
|
||||
|
||||
### 11.4 Handling “Unknown reachability”
|
||||
|
||||
Don’t force false certainty.
|
||||
|
||||
* Mark as `unknown` and show why (missing symbols, dynamic reflection, stripped binaries)
|
||||
* Policy can treat unknown as “reachable” for critical CVEs in internet‑facing services.
|
||||
|
||||
---
|
||||
|
||||
## 12) A concrete MVP path that still delivers value
|
||||
|
||||
If you want a minimal but real first release:
|
||||
|
||||
### MVP (2–3 deliverables)
|
||||
|
||||
1. **Evidence drawer** fed by:
|
||||
|
||||
* scanner output + SBOM + a simple “entrypoint map”
|
||||
2. **VEX workflow**
|
||||
|
||||
* allow engineers to set VEX status + justification
|
||||
3. **Signed decision gating**
|
||||
|
||||
* even if reachability is heuristic, the chain is real
|
||||
|
||||
Then iterate:
|
||||
|
||||
* add reachability graphs
|
||||
* add boundary extraction from IaC/K8s
|
||||
* tighten policy (staleness, confidence thresholds)
|
||||
|
||||
---
|
||||
|
||||
## 13) Quick checklist for “done enough to trust”
|
||||
|
||||
* [ ] Every finding expands to: Path, Boundary, VEX, Score, Proof
|
||||
* [ ] Every evidence tab shows `last_seen` + confidence
|
||||
* [ ] “Verify chain” works: SBOM → VEX → Decision all signed and bound to the artifact digest
|
||||
* [ ] Approve button signs a human approval attestation tied to the decision digest
|
||||
* [ ] CI gate verifies the same chain before deploy
|
||||
|
||||
---
|
||||
|
||||
If you want, I can also drop in:
|
||||
|
||||
* a full set of JSON Schemas for `stella/*@v1` predicates,
|
||||
* a reference verifier implementation outline in .NET 10 (Minimal API + a verifier class),
|
||||
* and a sample UI component tree (React) that renders path/boundary graphs and attestation chains.
|
||||
|
||||
[1]: https://github.com/secure-systems-lab/dsse?utm_source=chatgpt.com "DSSE: Dead Simple Signing Envelope"
|
||||
[2]: https://github.com/in-toto/attestation?utm_source=chatgpt.com "in-toto Attestation Framework"
|
||||
[3]: https://docs.sigstore.dev/about/bundle/?utm_source=chatgpt.com "Sigstore Bundle Format"
|
||||
[4]: https://spdx.dev/use/specifications/?utm_source=chatgpt.com "Specifications"
|
||||
[5]: https://github.com/CycloneDX/specification?utm_source=chatgpt.com "CycloneDX/specification"
|
||||
[6]: https://www.ntia.gov/sites/default/files/publications/vex_one-page_summary_0.pdf "VEX one-page summary"
|
||||
[7]: https://github.com/openvex/spec?utm_source=chatgpt.com "OpenVEX Specification"
|
||||
[8]: https://opencontainers.org/posts/blog/2024-03-13-image-and-distribution-1-1/?utm_source=chatgpt.com "OCI Image and Distribution Specs v1.1 Releases"
|
||||
[9]: https://oras.land/docs/concepts/reftypes/?utm_source=chatgpt.com "Attached Artifacts | OCI Registry As Storage"
|
||||
[10]: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-manage-artifact?utm_source=chatgpt.com "Manage OCI Artifacts and Supply Chain Artifacts with ORAS"
|
||||
[11]: https://openpolicyagent.org/docs/policy-language?utm_source=chatgpt.com "Policy Language"
|
||||
[12]: https://docs.sigstore.dev/cosign/verifying/attestation/?utm_source=chatgpt.com "In-Toto Attestations"
|
||||
@@ -0,0 +1,869 @@
|
||||
Here’s a compact, practical blueprint for bringing **EPSS** into your stack without chaos: a **3‑layer ingestion model** that keeps raw data, produces clean probabilities, and emits “signal‑ready” events your risk engine can use immediately.
|
||||
|
||||
---
|
||||
|
||||
# Why this matters (super short)
|
||||
|
||||
* **EPSS** = predicted probability a vuln will be exploited soon.
|
||||
* Mixing “raw EPSS feed” directly into decisions makes audits, rollbacks, and model upgrades painful.
|
||||
* A **layered model** lets you **version probability evolution**, compare vendors, and train **meta‑predictors on deltas** (how risk changes over time), not just on snapshots.
|
||||
|
||||
---
|
||||
|
||||
# The three layers (and how they map to Stella Ops)
|
||||
|
||||
1. **Raw feed layer (immutable)**
|
||||
|
||||
* **Goal:** Store exactly what the provider sent (EPSS v4 CSV/JSON, schema drift and all).
|
||||
* **Stella modules:** `Concelier` (preserve‑prune source) writes; `Authority` handles signatures/hashes.
|
||||
* **Storage:** `postgres.epss_raw` (partitioned by day); blob column for the untouched payload; SHA‑256 of source file.
|
||||
* **Why:** Full provenance + deterministic replay.
|
||||
|
||||
2. **Normalized probabilistic layer**
|
||||
|
||||
* **Goal:** Clean, typed tables keyed by `cve_id`, with **probability, percentile, model_version, asof_ts**.
|
||||
* **Stella modules:** `Excititor` (transform); `Policy Engine` reads.
|
||||
* **Storage:** `postgres.epss_prob` with a **surrogate key** `(cve_id, model_version, asof_ts)` and computed **delta fields** vs previous `asof_ts`.
|
||||
* **Extras:** Keep optional vendor columns (e.g., FIRST, custom regressors) to compare models side‑by‑side.
|
||||
|
||||
3. **Signal‑ready layer (risk engine contracts)**
|
||||
|
||||
* **Goal:** Pre‑chewed “events” your **Signals/Router** can route instantly.
|
||||
* **What’s inside:** Only the fields needed for gating and UI: `cve_id`, `prob_now`, `prob_delta`, `percentile`, `risk_band`, `explain_hash`.
|
||||
* **Emit:** `first_signal`, `risk_increase`, `risk_decrease`, `quieted` with **idempotent event keys**.
|
||||
* **Stella modules:** `Signals` publishes, `Router` fan‑outs, `Timeline` records; `Notify` handles subscriptions.
|
||||
|
||||
---
|
||||
|
||||
# Minimal Postgres schema (ready to paste)
|
||||
|
||||
```sql
|
||||
-- 1) Raw (immutable)
|
||||
create table epss_raw (
|
||||
id bigserial primary key,
|
||||
source_uri text not null,
|
||||
ingestion_ts timestamptz not null default now(),
|
||||
asof_date date not null,
|
||||
payload jsonb not null,
|
||||
payload_sha256 bytea not null
|
||||
);
|
||||
create index on epss_raw (asof_date);
|
||||
|
||||
-- 2) Normalized
|
||||
create table epss_prob (
|
||||
id bigserial primary key,
|
||||
cve_id text not null,
|
||||
model_version text not null, -- e.g., 'EPSS-4.0-Falcon-2025-12'
|
||||
asof_ts timestamptz not null,
|
||||
probability double precision not null,
|
||||
percentile double precision,
|
||||
features jsonb, -- optional: normalized features used
|
||||
unique (cve_id, model_version, asof_ts)
|
||||
);
|
||||
-- delta against prior point (materialized view or nightly job)
|
||||
create materialized view epss_prob_delta as
|
||||
select p.*,
|
||||
p.probability - lag(p.probability) over (partition by cve_id, model_version order by asof_ts) as prob_delta
|
||||
from epss_prob p;
|
||||
|
||||
-- 3) Signal-ready
|
||||
create table epss_signal (
|
||||
signal_id bigserial primary key,
|
||||
cve_id text not null,
|
||||
asof_ts timestamptz not null,
|
||||
probability double precision not null,
|
||||
prob_delta double precision,
|
||||
risk_band text not null, -- e.g., 'LOW/MED/HIGH/CRITICAL'
|
||||
model_version text not null,
|
||||
explain_hash bytea not null, -- hash of inputs -> deterministic
|
||||
unique (cve_id, model_version, asof_ts)
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# C# ingestion skeleton (StellaOps.Scanner.Worker.DotNet style)
|
||||
|
||||
```csharp
|
||||
// 1) Fetch & store raw (Concelier)
|
||||
public async Task IngestRawAsync(Uri src, DateOnly asOfDate) {
|
||||
var bytes = await http.GetByteArrayAsync(src);
|
||||
var sha = SHA256.HashData(bytes);
|
||||
await pg.ExecuteAsync(
|
||||
"insert into epss_raw(source_uri, asof_date, payload, payload_sha256) values (@u,@d,@p::jsonb,@s)",
|
||||
new { u = src.ToString(), d = asOfDate, p = Encoding.UTF8.GetString(bytes), s = sha });
|
||||
}
|
||||
|
||||
// 2) Normalize (Excititor)
|
||||
public async Task NormalizeAsync(DateOnly asOfDate, string modelVersion) {
|
||||
var raws = await pg.QueryAsync<(string Payload)>("select payload from epss_raw where asof_date=@d", new { d = asOfDate });
|
||||
foreach (var r in raws) {
|
||||
foreach (var row in ParseCsvOrJson(r.Payload)) {
|
||||
await pg.ExecuteAsync(
|
||||
@"insert into epss_prob(cve_id, model_version, asof_ts, probability, percentile, features)
|
||||
values (@cve,@mv,@ts,@prob,@pct,@feat)
|
||||
on conflict do nothing",
|
||||
new { cve = row.Cve, mv = modelVersion, ts = row.AsOf, prob = row.Prob, pct = row.Pctl, feat = row.Features });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3) Emit signal-ready (Signals)
|
||||
public async Task EmitSignalsAsync(string modelVersion, double deltaThreshold) {
|
||||
var rows = await pg.QueryAsync(@"select cve_id, asof_ts, probability,
|
||||
probability - lag(probability) over (partition by cve_id, model_version order by asof_ts) as prob_delta
|
||||
from epss_prob where model_version=@mv", new { mv = modelVersion });
|
||||
|
||||
foreach (var r in rows) {
|
||||
var band = Band(r.probability); // map to LOW/MED/HIGH/CRITICAL
|
||||
if (Math.Abs(r.prob_delta ?? 0) >= deltaThreshold) {
|
||||
var explainHash = DeterministicExplainHash(r);
|
||||
await pg.ExecuteAsync(@"insert into epss_signal
|
||||
(cve_id, asof_ts, probability, prob_delta, risk_band, model_version, explain_hash)
|
||||
values (@c,@t,@p,@d,@b,@mv,@h)
|
||||
on conflict do nothing",
|
||||
new { c = r.cve_id, t = r.asof_ts, p = r.probability, d = r.prob_delta, b = band, mv = modelVersion, h = explainHash });
|
||||
|
||||
await bus.PublishAsync("risk.epss.delta", new {
|
||||
cve = r.cve_id, ts = r.asof_ts, prob = r.probability, delta = r.prob_delta, band, model = modelVersion, explain = Convert.ToHexString(explainHash)
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Versioning & experiments (the secret sauce)
|
||||
|
||||
* **Model namespace:** `EPSS‑4.0‑<regressor‑name>‑<date>` so you can run multiple variants in parallel.
|
||||
* **Delta‑training:** Train a small meta‑predictor on **Δprobability** to forecast **“risk jumps in next N days.”**
|
||||
* **A/B in production:** Route `model_version=x` to 50% of projects; compare **MTTA to patch** and **false‑alarm rate**.
|
||||
|
||||
---
|
||||
|
||||
# Policy & UI wiring (quick contracts)
|
||||
|
||||
**Policy gates** (OPA/Rego or internal rules):
|
||||
|
||||
* Block if `risk_band ∈ {HIGH, CRITICAL}` **AND** `prob_delta >= 0.1` in last 72h.
|
||||
* Soften if asset not reachable or mitigated by VEX.
|
||||
|
||||
**UI (Evidence pane):**
|
||||
|
||||
* Show **sparkline of EPSS over time**, highlight last Δ.
|
||||
* “Why now?” button reveals **explain_hash** → deterministic evidence payload.
|
||||
|
||||
---
|
||||
|
||||
# Ops & reliability
|
||||
|
||||
* Daily ingestion with **idempotent** runs (raw SHA guard).
|
||||
* Backfills: re‑normalize from `epss_raw` for any new model without re‑downloading.
|
||||
* **Deterministic replay:** export `(raw, transform code hash, model_version)` alongside results.
|
||||
|
||||
---
|
||||
|
||||
If you want, I can drop this as a ready‑to‑run **.sql + .csproj** seed with a tiny CLI (`ingest`, `normalize`, `emit`) tailored to your `Postgres + Valkey` profile.
|
||||
Below is a “do this, then this” implementation guide for a **layered EPSS pipeline** inside **Stella Ops**, with concrete schemas, job boundaries, idempotency rules, and the tricky edge cases (model-version shifts, noise control, backfills).
|
||||
|
||||
I’ll assume:
|
||||
|
||||
* **Postgres** is your system of record, **Valkey** is available for caching,
|
||||
* you run **.NET workers** (like `StellaOps.Scanner.Worker.DotNet`),
|
||||
* Stella modules you referenced map roughly like this:
|
||||
|
||||
* **Concelier** = ingest + preserve/prune raw sources
|
||||
* **Authority** = provenance (hashes, immutability, signature-like guarantees)
|
||||
* **Excititor** = transform/normalize
|
||||
* **Signals / Router / Timeline / Notify** = event pipeline + audit trail + subscriptions
|
||||
|
||||
I’ll anchor the EPSS feed details to FIRST’s docs:
|
||||
|
||||
* The data feed fields are `cve`, `epss`, `percentile` and are refreshed daily. ([FIRST][1])
|
||||
* Historical daily `.csv.gz` files exist at `https://epss.empiricalsecurity.com/epss_scores-YYYY-mm-dd.csv.gz`. ([FIRST][1])
|
||||
* The API base is `https://api.first.org/data/v1/epss` and supports per-CVE and time-series queries. ([FIRST][2])
|
||||
* FIRST notes model-version shifts (v2/v3/v4) and that the daily files include a leading `#` comment indicating model version/publish date (important for delta correctness). ([FIRST][1])
|
||||
* FIRST’s guidance: use **probability** as the primary score and **show percentile alongside it**; raw feeds provide both as decimals 0–1. ([FIRST][3])
|
||||
|
||||
---
|
||||
|
||||
## 0) Target architecture and data contracts
|
||||
|
||||
### The 3 layers and what must be true in each
|
||||
|
||||
1. **Raw layer (immutable)**
|
||||
|
||||
* You can replay exactly what you ingested, byte-for-byte.
|
||||
* Contains: file bytes or object-store pointer, headers (ETag, Last-Modified), SHA-256, parsed “header comment” (the `# …` line), ingestion status.
|
||||
|
||||
2. **Normalized probability layer (typed, queryable, historical)**
|
||||
|
||||
* One row per `(model_name, asof_date, cve_id)`.
|
||||
* Contains: `epss` probability (0–1), `percentile` (0–1), `model_version` (from file header comment if available).
|
||||
* Built for joins against vulnerability inventory and for time series.
|
||||
|
||||
3. **Signal-ready layer (risk engine contract)**
|
||||
|
||||
* Contains only actionable changes (crossing thresholds, jumps, newly-scored, etc.), ideally scoped to **observed CVEs** in your environment to avoid noise.
|
||||
* Events are idempotent, audit-friendly, and versioned.
|
||||
|
||||
---
|
||||
|
||||
## 1) Data source choice and acquisition strategy
|
||||
|
||||
### Prefer the daily bulk `.csv.gz` over paging the API for full refresh
|
||||
|
||||
* FIRST explicitly documents the “ALL CVEs for a date” bulk file URL pattern. ([FIRST][2])
|
||||
* The API is great for:
|
||||
|
||||
* “give me EPSS for this CVE list”
|
||||
* “give me last 30 days time series for CVE X” ([FIRST][2])
|
||||
|
||||
**Recommendation**
|
||||
|
||||
* Daily job pulls the bulk file for “latest available date”.
|
||||
* A separate on-demand endpoint uses the API time-series for UI convenience (optional).
|
||||
|
||||
### Robust “latest available date” probing
|
||||
|
||||
Because the “current day” file may not be published when your cron fires:
|
||||
|
||||
Algorithm:
|
||||
|
||||
1. Let `d0 = UtcToday`.
|
||||
2. For `d in [d0, d0-1, d0-2, d0-3]`:
|
||||
|
||||
* Try `GET https://epss.empiricalsecurity.com/epss_scores-{d:yyyy-MM-dd}.csv.gz`
|
||||
* If HTTP 200: ingest that as `asof_date = d` and stop.
|
||||
3. If none succeed: fail the job with a clear message + alert.
|
||||
|
||||
This avoids timezone and publishing-time ambiguity.
|
||||
|
||||
---
|
||||
|
||||
## 2) Layer 1: Raw feed (Concelier + Authority)
|
||||
|
||||
### 2.1 Schema for raw + lineage
|
||||
|
||||
Use a dedicated schema `epss` so the pipeline is easy to reason about.
|
||||
|
||||
```sql
|
||||
create schema if not exists epss;
|
||||
|
||||
-- Immutable file-level record
|
||||
create table if not exists epss.raw_file (
|
||||
raw_id bigserial primary key,
|
||||
source_uri text not null,
|
||||
asof_date date not null,
|
||||
fetched_at timestamptz not null default now(),
|
||||
|
||||
http_etag text,
|
||||
http_last_modified timestamptz,
|
||||
content_len bigint,
|
||||
|
||||
content_sha256 bytea not null,
|
||||
|
||||
-- first non-empty comment lines like "# model=... date=..."
|
||||
header_comment text,
|
||||
model_version text,
|
||||
model_published_on date,
|
||||
|
||||
-- storage: either inline bytea OR object storage pointer
|
||||
storage_kind text not null default 'pg_bytea', -- 'pg_bytea' | 's3' | 'fs'
|
||||
storage_ref text,
|
||||
content_gz bytea, -- nullable if stored externally
|
||||
|
||||
parse_status text not null default 'pending', -- pending|parsed|failed
|
||||
parse_error text,
|
||||
|
||||
unique (source_uri, asof_date, content_sha256)
|
||||
);
|
||||
|
||||
create index if not exists ix_epss_raw_file_asof on epss.raw_file(asof_date);
|
||||
create index if not exists ix_epss_raw_file_status on epss.raw_file(parse_status);
|
||||
```
|
||||
|
||||
**Why store `model_version` here?**
|
||||
FIRST warns that model updates cause “major shifts” and the daily files include a `#` comment with model version/publish date. If you ignore this, your delta logic will misfire on model-change days. ([FIRST][1])
|
||||
|
||||
### 2.2 Raw ingestion idempotency rules
|
||||
|
||||
A run is “already ingested” if:
|
||||
|
||||
* a row exists for `(source_uri, asof_date)` with the same `content_sha256`, OR
|
||||
* you implement “single truth per day” and treat any new sha for the same date as “replace” (rare, but can happen).
|
||||
|
||||
Recommended:
|
||||
|
||||
* **Treat as replace only if** you’re confident the source can republish the same date. If not, keep both but mark the superseded one.
|
||||
|
||||
### 2.3 Raw ingestion implementation details (.NET)
|
||||
|
||||
**Key constraints**
|
||||
|
||||
* Download as a stream (`ResponseHeadersRead`)
|
||||
* Compute SHA-256 while streaming
|
||||
* Store bytes or stream them into object storage
|
||||
* Capture ETag/Last-Modified headers if present
|
||||
|
||||
Pseudo-implementation structure:
|
||||
|
||||
* `EpssFetchJob`
|
||||
|
||||
* `ProbeLatestDateAsync()`
|
||||
* `DownloadAsync(uri)`
|
||||
* `ExtractHeaderCommentAsync(gzipStream)` (read a few first lines after decompression)
|
||||
* `InsertRawFileRecord(...)` (Concelier + Authority)
|
||||
|
||||
**Header comment extraction**
|
||||
FIRST indicates files may start with `# ... model version ... publish date ...`. ([FIRST][1])
|
||||
So do:
|
||||
|
||||
* Decompress
|
||||
* Read lines until you find first non-empty non-`#` line (that’s likely CSV header / first row)
|
||||
* Save the concatenated `#` lines as `header_comment`
|
||||
* Regex best-effort parse:
|
||||
|
||||
* `model_version`: something like `v2025.03.14`
|
||||
* `model_published_on`: `YYYY-MM-DD`
|
||||
|
||||
If parsing fails, still store `header_comment`.
|
||||
|
||||
### 2.4 Pruning raw (Concelier “preserve-prune”)
|
||||
|
||||
Define retention policy:
|
||||
|
||||
* Keep **raw bytes** 90–180 days (cheap enough; each `.csv.gz` is usually a few–tens of MB)
|
||||
* Keep **metadata** forever (tiny, essential for audits)
|
||||
|
||||
Nightly cleanup job:
|
||||
|
||||
* delete `content_gz` or external object for `raw_file` older than retention
|
||||
* keep row but set `storage_kind='pruned'`, `content_gz=null`, `storage_ref=null`
|
||||
|
||||
---
|
||||
|
||||
## 3) Layer 2: Normalized probability tables (Excititor)
|
||||
|
||||
### 3.1 Core normalized table design
|
||||
|
||||
Requirements:
|
||||
|
||||
* Efficient time series per CVE
|
||||
* Efficient “latest score per CVE”
|
||||
* Efficient join to “observed vulnerabilities” tables
|
||||
|
||||
#### Daily score table (partitioned)
|
||||
|
||||
```sql
|
||||
create table if not exists epss.daily_score (
|
||||
model_name text not null, -- 'FIRST_EPSS'
|
||||
asof_date date not null,
|
||||
cve_id text not null,
|
||||
epss double precision not null,
|
||||
percentile double precision,
|
||||
model_version text, -- from raw header if available
|
||||
raw_id bigint references epss.raw_file(raw_id),
|
||||
loaded_at timestamptz not null default now(),
|
||||
|
||||
-- Guards
|
||||
constraint ck_epss_range check (epss >= 0.0 and epss <= 1.0),
|
||||
constraint ck_percentile_range check (percentile is null or (percentile >= 0.0 and percentile <= 1.0)),
|
||||
|
||||
primary key (model_name, asof_date, cve_id)
|
||||
) partition by range (asof_date);
|
||||
|
||||
-- Example monthly partitions (create via migration script generator)
|
||||
create table if not exists epss.daily_score_2025_12
|
||||
partition of epss.daily_score for values from ('2025-12-01') to ('2026-01-01');
|
||||
|
||||
create index if not exists ix_epss_daily_score_cve on epss.daily_score (model_name, cve_id, asof_date desc);
|
||||
create index if not exists ix_epss_daily_score_epss on epss.daily_score (model_name, asof_date, epss desc);
|
||||
create index if not exists ix_epss_daily_score_pct on epss.daily_score (model_name, asof_date, percentile desc);
|
||||
```
|
||||
|
||||
**Field semantics**
|
||||
|
||||
* `epss` is the probability of exploitation in the next 30 days, 0–1. ([FIRST][1])
|
||||
* `percentile` is relative rank among all scored vulnerabilities. ([FIRST][1])
|
||||
|
||||
### 3.2 Maintain a “latest” table for fast joins
|
||||
|
||||
Don’t compute latest via window functions in hot paths (policy evaluation / scoring). Materialize it.
|
||||
|
||||
```sql
|
||||
create table if not exists epss.latest_score (
|
||||
model_name text not null,
|
||||
cve_id text not null,
|
||||
asof_date date not null,
|
||||
epss double precision not null,
|
||||
percentile double precision,
|
||||
model_version text,
|
||||
updated_at timestamptz not null default now(),
|
||||
primary key (model_name, cve_id)
|
||||
);
|
||||
|
||||
create index if not exists ix_epss_latest_epss on epss.latest_score(model_name, epss desc);
|
||||
create index if not exists ix_epss_latest_pct on epss.latest_score(model_name, percentile desc);
|
||||
```
|
||||
|
||||
Update logic (after loading a day):
|
||||
|
||||
* Upsert each CVE (or do a set-based upsert):
|
||||
|
||||
* `asof_date` should only move forward
|
||||
* if a backfill loads an older day, do not overwrite latest
|
||||
|
||||
### 3.3 Delta table for change detection
|
||||
|
||||
Store deltas per day (this powers signals and “sparkline deltas”).
|
||||
|
||||
```sql
|
||||
create table if not exists epss.daily_delta (
|
||||
model_name text not null,
|
||||
asof_date date not null,
|
||||
cve_id text not null,
|
||||
|
||||
epss double precision not null,
|
||||
prev_asof_date date,
|
||||
prev_epss double precision,
|
||||
epss_delta double precision,
|
||||
|
||||
percentile double precision,
|
||||
prev_percentile double precision,
|
||||
percentile_delta double precision,
|
||||
|
||||
model_version text,
|
||||
prev_model_version text,
|
||||
is_model_change boolean not null default false,
|
||||
|
||||
created_at timestamptz not null default now(),
|
||||
primary key (model_name, asof_date, cve_id)
|
||||
);
|
||||
|
||||
create index if not exists ix_epss_daily_delta_cve on epss.daily_delta(model_name, cve_id, asof_date desc);
|
||||
create index if not exists ix_epss_daily_delta_delta on epss.daily_delta(model_name, asof_date, epss_delta desc);
|
||||
```
|
||||
|
||||
**Model update handling**
|
||||
|
||||
* On a model version change day (v3→v4 etc), many deltas will jump.
|
||||
* FIRST explicitly warns model shifts. ([FIRST][1])
|
||||
So:
|
||||
* detect if today’s `model_version != previous_day.model_version`
|
||||
* set `is_model_change = true`
|
||||
* optionally **suppress delta-based signals** that day (or emit a separate “MODEL_UPDATED” event)
|
||||
|
||||
### 3.4 Normalization job mechanics
|
||||
|
||||
Implement `EpssNormalizeJob`:
|
||||
|
||||
1. Select `raw_file` rows where `parse_status='pending'`.
|
||||
2. Decompress `content_gz` or fetch from object store.
|
||||
3. Parse CSV:
|
||||
|
||||
* skip `#` comment lines
|
||||
* expect columns: `cve,epss,percentile` (FIRST documents these fields). ([FIRST][1])
|
||||
4. Validate:
|
||||
|
||||
* CVE format: `^CVE-\d{4}-\d{4,}$`
|
||||
* numeric parse for epss/percentile
|
||||
* range checks 0–1
|
||||
5. Load into Postgres fast:
|
||||
|
||||
* Use `COPY` (binary import) into a **staging table** `epss.stage_score`
|
||||
* Then set-based insert into `epss.daily_score`
|
||||
6. Update `epss.raw_file.parse_status='parsed'` or `failed`.
|
||||
|
||||
#### Staging table pattern
|
||||
|
||||
```sql
|
||||
create unlogged table if not exists epss.stage_score (
|
||||
model_name text not null,
|
||||
asof_date date not null,
|
||||
cve_id text not null,
|
||||
epss double precision not null,
|
||||
percentile double precision,
|
||||
model_version text,
|
||||
raw_id bigint not null
|
||||
);
|
||||
```
|
||||
|
||||
In the job:
|
||||
|
||||
* `truncate epss.stage_score;`
|
||||
* `COPY epss.stage_score FROM STDIN (FORMAT BINARY)`
|
||||
* Then (transactionally):
|
||||
|
||||
* `delete from epss.daily_score where model_name=@m and asof_date=@d;` *(idempotency for reruns)*
|
||||
* `insert into epss.daily_score (...) select ... from epss.stage_score;`
|
||||
|
||||
This avoids `ON CONFLICT` overhead and guarantees deterministic reruns.
|
||||
|
||||
### 3.5 Delta + latest materialization job
|
||||
|
||||
Implement `EpssMaterializeJob` after successful daily_score insert.
|
||||
|
||||
**Compute previous available date**
|
||||
|
||||
```sql
|
||||
-- previous date available for that model_name
|
||||
select max(asof_date)
|
||||
from epss.daily_score
|
||||
where model_name = @model
|
||||
and asof_date < @asof_date;
|
||||
```
|
||||
|
||||
**Populate delta (set-based)**
|
||||
|
||||
```sql
|
||||
insert into epss.daily_delta (
|
||||
model_name, asof_date, cve_id,
|
||||
epss, prev_asof_date, prev_epss, epss_delta,
|
||||
percentile, prev_percentile, percentile_delta,
|
||||
model_version, prev_model_version, is_model_change
|
||||
)
|
||||
select
|
||||
cur.model_name,
|
||||
cur.asof_date,
|
||||
cur.cve_id,
|
||||
cur.epss,
|
||||
prev.asof_date as prev_asof_date,
|
||||
prev.epss as prev_epss,
|
||||
cur.epss - prev.epss as epss_delta,
|
||||
cur.percentile,
|
||||
prev.percentile as prev_percentile,
|
||||
(cur.percentile - prev.percentile) as percentile_delta,
|
||||
cur.model_version,
|
||||
prev.model_version,
|
||||
(cur.model_version is not null and prev.model_version is not null and cur.model_version <> prev.model_version) as is_model_change
|
||||
from epss.daily_score cur
|
||||
left join epss.daily_score prev
|
||||
on prev.model_name = cur.model_name
|
||||
and prev.asof_date = @prev_asof_date
|
||||
and prev.cve_id = cur.cve_id
|
||||
where cur.model_name = @model
|
||||
and cur.asof_date = @asof_date;
|
||||
```
|
||||
|
||||
**Update latest_score (set-based upsert)**
|
||||
|
||||
```sql
|
||||
insert into epss.latest_score(model_name, cve_id, asof_date, epss, percentile, model_version)
|
||||
select model_name, cve_id, asof_date, epss, percentile, model_version
|
||||
from epss.daily_score
|
||||
where model_name=@model and asof_date=@asof_date
|
||||
on conflict (model_name, cve_id) do update
|
||||
set asof_date = excluded.asof_date,
|
||||
epss = excluded.epss,
|
||||
percentile = excluded.percentile,
|
||||
model_version = excluded.model_version,
|
||||
updated_at = now()
|
||||
where epss.latest_score.asof_date < excluded.asof_date;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4) Layer 3: Signal-ready output (Signals + Router + Timeline + Notify)
|
||||
|
||||
### 4.1 Decide what “signal” means in Stella Ops
|
||||
|
||||
You do **not** want to emit 300k events daily.
|
||||
|
||||
You want “actionable” events, ideally:
|
||||
|
||||
* only for CVEs that are **observed** in your tenant’s environment, and
|
||||
* only when something meaningful happens.
|
||||
|
||||
Examples:
|
||||
|
||||
* Risk band changes (based on percentile or probability)
|
||||
* ΔEPS S crosses a threshold (e.g., jump ≥ 0.05)
|
||||
* Newly scored CVEs that are present in environment
|
||||
* Model version change day → one summary event instead of 300k deltas
|
||||
|
||||
### 4.2 Risk band mapping (internal heuristic)
|
||||
|
||||
FIRST explicitly does **not** “officially bin” EPSS scores; binning is subjective. ([FIRST][3])
|
||||
But operationally you’ll want bands. Use config-driven thresholds.
|
||||
|
||||
Default band function based on percentile:
|
||||
|
||||
* `CRITICAL` if `percentile >= 0.995`
|
||||
* `HIGH` if `percentile >= 0.99`
|
||||
* `MEDIUM` if `percentile >= 0.90`
|
||||
* else `LOW`
|
||||
|
||||
Store these in config per tenant/policy pack.
|
||||
|
||||
### 4.3 Signal table for idempotency + audit
|
||||
|
||||
```sql
|
||||
create table if not exists epss.signal (
|
||||
signal_id bigserial primary key,
|
||||
tenant_id uuid not null,
|
||||
model_name text not null,
|
||||
asof_date date not null,
|
||||
cve_id text not null,
|
||||
|
||||
event_type text not null, -- 'RISK_BAND_UP' | 'RISK_SPIKE' | 'MODEL_UPDATED' | ...
|
||||
risk_band text,
|
||||
epss double precision,
|
||||
epss_delta double precision,
|
||||
percentile double precision,
|
||||
percentile_delta double precision,
|
||||
|
||||
is_model_change boolean not null default false,
|
||||
|
||||
-- deterministic idempotency key
|
||||
dedupe_key text not null,
|
||||
payload jsonb not null,
|
||||
|
||||
created_at timestamptz not null default now(),
|
||||
|
||||
unique (tenant_id, dedupe_key)
|
||||
);
|
||||
|
||||
create index if not exists ix_epss_signal_tenant_date on epss.signal(tenant_id, asof_date desc);
|
||||
create index if not exists ix_epss_signal_cve on epss.signal(tenant_id, cve_id, asof_date desc);
|
||||
```
|
||||
|
||||
**Dedupe key pattern**
|
||||
Make it deterministic:
|
||||
|
||||
```
|
||||
dedupe_key = $"{model_name}:{asof_date:yyyy-MM-dd}:{cve_id}:{event_type}:{band_before}->{band_after}"
|
||||
```
|
||||
|
||||
### 4.4 Signal generation job
|
||||
|
||||
Implement `EpssSignalJob(tenant)`:
|
||||
|
||||
1. Get tenant’s **observed CVEs** from your vuln inventory (whatever your table is; call it `vuln.instance`):
|
||||
|
||||
* only open/unremediated vulns
|
||||
* optionally only “reachable” or “internet exposed” assets
|
||||
|
||||
2. Join against today’s `epss.daily_delta` (or `epss.daily_score` if you skipped delta):
|
||||
|
||||
Pseudo-SQL:
|
||||
|
||||
```sql
|
||||
select d.*
|
||||
from epss.daily_delta d
|
||||
join vuln.observed_cve oc
|
||||
on oc.tenant_id = @tenant
|
||||
and oc.cve_id = d.cve_id
|
||||
where d.model_name=@model
|
||||
and d.asof_date=@asof_date;
|
||||
```
|
||||
|
||||
3. Suppress noise:
|
||||
|
||||
* if `is_model_change=true`, skip “delta spike” events and instead emit one `MODEL_UPDATED` summary event per tenant (and maybe per policy domain).
|
||||
* else evaluate:
|
||||
|
||||
* `abs(epss_delta) >= delta_threshold`
|
||||
* band change
|
||||
* percentile crosses a cutoff
|
||||
|
||||
4. Insert into `epss.signal` with dedupe key, then publish to Signals bus:
|
||||
|
||||
* topic: `signals.epss`
|
||||
* payload includes `tenant_id`, `cve_id`, `asof_date`, `epss`, `percentile`, deltas, band, and an `evidence` block.
|
||||
|
||||
5. Timeline + Notify:
|
||||
|
||||
* Timeline: record the event (what changed, when, data source sha)
|
||||
* Notify: notify subscribed channels (Slack/email/etc) based on tenant policy
|
||||
|
||||
### 4.5 Evidence payload structure
|
||||
|
||||
Keep evidence deterministic + replayable:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": {
|
||||
"provider": "FIRST",
|
||||
"feed": "epss_scores-YYYY-MM-DD.csv.gz",
|
||||
"asof_date": "2025-12-17",
|
||||
"raw_sha256": "…",
|
||||
"model_version": "v2025.03.14",
|
||||
"header_comment": "# ... "
|
||||
},
|
||||
"metrics": {
|
||||
"epss": 0.153,
|
||||
"percentile": 0.92,
|
||||
"epss_delta": 0.051,
|
||||
"percentile_delta": 0.03
|
||||
},
|
||||
"decision": {
|
||||
"event_type": "RISK_SPIKE",
|
||||
"thresholds": {
|
||||
"delta_threshold": 0.05,
|
||||
"critical_percentile": 0.995
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This aligns with FIRST’s recommendation to present probability with percentile when possible. ([FIRST][3])
|
||||
|
||||
---
|
||||
|
||||
## 5) Integration points inside Stella Ops
|
||||
|
||||
### 5.1 Policy Engine usage
|
||||
|
||||
Policy Engine should **only** read from Layer 2 (normalized) and Layer 3 (signals), never raw.
|
||||
|
||||
Patterns:
|
||||
|
||||
* For gating decisions: query `epss.latest_score` for each CVE in a build/image/SBOM scan result.
|
||||
* For “why was this blocked?”: show evidence that references `raw_sha256` and `model_version`.
|
||||
|
||||
### 5.2 Vuln scoring pipeline
|
||||
|
||||
When you compute “Stella Risk Score” for a vuln instance:
|
||||
|
||||
* Join `vuln_instance.cve_id` → `epss.latest_score`
|
||||
* Combine with CVSS, KEV, exploit maturity, asset exposure, etc.
|
||||
* EPSS alone is **threat likelihood**, not impact; FIRST explicitly says it’s not a complete picture of risk. ([FIRST][4])
|
||||
|
||||
### 5.3 UI display
|
||||
|
||||
Recommended UI string (per FIRST guidance):
|
||||
|
||||
* Show **probability** as a percent + show percentile:
|
||||
|
||||
* `15.3% (92nd percentile)` ([FIRST][3])
|
||||
|
||||
For sparklines:
|
||||
|
||||
* Use `epss.daily_score` time series for last N days
|
||||
* Annotate model-version change days (vertical marker)
|
||||
|
||||
---
|
||||
|
||||
## 6) Operational hardening
|
||||
|
||||
### 6.1 Scheduling
|
||||
|
||||
* Run daily at a fixed time in UTC.
|
||||
* Probe up to 3 back days for latest file.
|
||||
|
||||
### 6.2 Exactly-once semantics
|
||||
|
||||
Use three safeguards:
|
||||
|
||||
1. `epss.raw_file` uniqueness on `(source_uri, asof_date, sha256)`
|
||||
2. Transactional load:
|
||||
|
||||
* delete existing `daily_score` for that `(model_name, asof_date)`
|
||||
* insert freshly parsed rows
|
||||
3. Advisory lock per `(model_name, asof_date)` to prevent concurrent loads:
|
||||
|
||||
* `pg_advisory_xact_lock(hashtext(model_name), asof_date::int)`
|
||||
|
||||
### 6.3 Monitoring (must-have metrics)
|
||||
|
||||
Emit metrics per job stage:
|
||||
|
||||
* download success/failure
|
||||
* bytes downloaded
|
||||
* sha256 computed
|
||||
* rows parsed
|
||||
* parse error count
|
||||
* rows inserted into `daily_score`
|
||||
* delta rows created
|
||||
* signal events emitted
|
||||
* “model version changed” boolean
|
||||
|
||||
Alert conditions:
|
||||
|
||||
* no new asof_date ingested for > 48 hours
|
||||
* parse failure
|
||||
* row count drops by > X% from previous day (data anomaly)
|
||||
|
||||
### 6.4 Backfills
|
||||
|
||||
Implement `epss backfill --from 2021-04-14 --to 2025-12-17`:
|
||||
|
||||
* Fetch raw files for each day
|
||||
* Normalize daily_score
|
||||
* Materialize latest and delta
|
||||
* **Disable signals** during bulk backfill (or route to “silent” topic) to avoid spamming.
|
||||
|
||||
FIRST notes historical data begins 2021-04-14. ([FIRST][1])
|
||||
|
||||
---
|
||||
|
||||
## 7) Reference .NET job skeletons
|
||||
|
||||
### Job boundaries
|
||||
|
||||
* `EpssFetchJob` → writes `epss.raw_file`
|
||||
* `EpssNormalizeJob` → fills `epss.daily_score`
|
||||
* `EpssMaterializeJob` → updates `epss.daily_delta` and `epss.latest_score`
|
||||
* `EpssSignalJob` → per-tenant emission into `epss.signal` + bus publish
|
||||
|
||||
### Performance notes
|
||||
|
||||
* Use `GZipStream` + `StreamReader` line-by-line (no full file into memory)
|
||||
* Use `NpgsqlBinaryImporter` for `COPY` into staging
|
||||
* Use set-based SQL for delta/latest
|
||||
|
||||
---
|
||||
|
||||
## 8) The “gotchas” that make or break EPSS pipelines
|
||||
|
||||
1. **Model version changes create false deltas**
|
||||
Store `model_version` and mark `is_model_change`. FIRST explicitly warns about score shifts on model updates and notes v4 began publishing on 2025‑03‑17. ([FIRST][1])
|
||||
|
||||
2. **Percentile is relative; probability is primary**
|
||||
Probability should remain your canonical numeric score; percentile provides context for humans. ([FIRST][3])
|
||||
|
||||
3. **Don’t emit global events**
|
||||
Restrict signals to observed CVEs per tenant/environment.
|
||||
|
||||
4. **Keep raw provenance**
|
||||
Your audit story depends on storing:
|
||||
|
||||
* exact source URI, as-of date, sha256, header comment
|
||||
|
||||
---
|
||||
|
||||
## 9) Minimal “definition of done” checklist
|
||||
|
||||
### Data correctness
|
||||
|
||||
* [ ] For a known CVE, `epss.latest_score` matches the daily file for the latest asof_date
|
||||
* [ ] `epss` and `percentile` ranges enforced (0–1)
|
||||
* [ ] Model version extracted when present; otherwise stored as null but header_comment preserved
|
||||
* [ ] Delta rows created and `is_model_change` flips on version changes
|
||||
|
||||
### Operational
|
||||
|
||||
* [ ] Daily job retries on transient HTTP failures
|
||||
* [ ] Alert if no new asof_date in 48h
|
||||
* [ ] Raw bytes retention + metadata retention
|
||||
|
||||
### Product
|
||||
|
||||
* [ ] UI displays `probability% (percentile)` per FIRST recommendation ([FIRST][3])
|
||||
* [ ] Signal events link to evidence (raw sha, model version, asof date)
|
||||
* [ ] Policy Engine consumes `latest_score` only (never raw)
|
||||
|
||||
---
|
||||
|
||||
If you want, I can also provide:
|
||||
|
||||
* a **migration script generator** that auto-creates monthly partitions for `epss.daily_score`,
|
||||
* an example **Valkey caching strategy** (`epss:latest:{cve}` with a 48h TTL, warmed only for observed CVEs),
|
||||
* and a concrete **“observed CVE” join contract** (what columns to expose from your vuln inventory so EPSS signals stay noise-free).
|
||||
|
||||
[1]: https://www.first.org/epss/data_stats "Exploit Prediction Scoring System (EPSS)"
|
||||
[2]: https://www.first.org/epss/api "Exploit Prediction Scoring System (EPSS)"
|
||||
[3]: https://www.first.org/epss/articles/prob_percentile_bins "Exploit Prediction Scoring System (EPSS)"
|
||||
[4]: https://www.first.org/epss/faq "EPSS Frequently Asked Questions"
|
||||
@@ -88,6 +88,8 @@ Detects configuration-based gates:
|
||||
|
||||
### DetectedGate
|
||||
|
||||
**Note:** In **Signals API outputs**, `type` is serialized as the C# enum name (e.g., `"AuthRequired"`). In **richgraph-v1** JSON, `type` is lowerCamelCase and gate fields are snake_case (see example below).
|
||||
|
||||
```typescript
|
||||
interface DetectedGate {
|
||||
type: 'AuthRequired' | 'FeatureFlag' | 'AdminOnly' | 'NonDefaultConfig';
|
||||
@@ -130,6 +132,25 @@ public sealed record RichGraphEdge
|
||||
}
|
||||
```
|
||||
|
||||
**richgraph-v1 JSON example (edge fragment):**
|
||||
|
||||
```json
|
||||
{
|
||||
"gate_multiplier_bps": 3000,
|
||||
"gates": [
|
||||
{
|
||||
"type": "authRequired",
|
||||
"detail": "[Authorize] attribute on controller",
|
||||
"guard_symbol": "MyController.VulnerableAction",
|
||||
"source_file": "src/MyController.cs",
|
||||
"line_number": 42,
|
||||
"detection_method": "csharp.attribute",
|
||||
"confidence": 0.95
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### ReachabilityReport
|
||||
|
||||
Gates are included in the reachability report:
|
||||
|
||||
Reference in New Issue
Block a user