release orchestrator v1 draft and build fixes

2026-01-12 12:24:17 +02:00
parent f3de858c59
commit 9873f80830
1598 changed files with 240385 additions and 5944 deletions
--- a/docs-archived/sprints/evid-001/EVIDENCE_PIPELINE_CONSOLIDATION.md
+++ b/docs-archived/sprints/evid-001/EVIDENCE_PIPELINE_CONSOLIDATION.md
@@ -0,0 +1,487 @@
+# Evidence Pipeline Consolidation - Sprint Plan
+
+**Created**: 2026-01-11
+**Status**: Planning
+**Owner**: Stella Ops Engineering
+
+---
+
+## Executive Summary
+
+This document consolidates the product advisory for "reachability + runtime + patch-diff" evidence into actionable sprints. **Critical finding: 70-80% of the proposed infrastructure already exists in Stella Ops.** The work is primarily about:
+
+1. **Connecting** existing components
+2. **Implementing** missing adapters (runtime capture, binary diff)
+3. **Orchestrating** CVE-to-verdict flow
+4. **Enhancing** evidence-to-VEX integration
+
+---
+
+## Existing Infrastructure Analysis
+
+### Already Implemented (No New Development Needed)
+
+| Component | Location | Status |
+|-----------|----------|--------|
+| **ReachabilityAnalyzer** | `Scanner.CallGraph/Analysis/` | Complete - BFS traversal, deterministic paths |
+| **ReachabilityLattice** | `Scanner.Emit/Reachability/` | Complete - Score-based verdict merge |
+| **ReachabilityStackEvaluator** | `Scanner.Reachability/Stack/` | Complete - 3-layer evaluation |
+| **ReachabilityWitnessDsseBuilder** | `Scanner.Reachability/Attestation/` | Complete - in-toto statements |
+| **ReachabilityEvidence** | `Evidence.Bundle/` | Complete - FunctionPath, ImportChain |
+| **ReachabilityResult** | `Scanner.Reachability/Witnesses/` | Complete - PathWitness/SuppressionWitness |
+| **EvidenceBundle** | `Evidence.Bundle/` | Complete - Multi-type evidence container |
+| **EvidenceDbContext** | `Evidence.Persistence/` | Complete - Postgres persistence |
+| **RuntimeEvidence models** | `Scanner.Analyzers.Native/RuntimeCapture/` | Complete - Session, LoadEvent, Edge |
+| **DotNetCallGraphExtractor** | `Scanner.CallGraph/Extraction/DotNet/` | Complete - .NET call graphs |
+| **DotNetReachabilityLifter** | `Scanner.Reachability/Lifters/` | Complete - Lift to union model |
+| **CVE-Symbol mapping schema** | `devops/database/migrations/` | Complete - Tables and indexes |
+| **DSSE signing** | `Scanner.Worker/Processing/Surface/` | Complete - IDsseEnvelopeSigner |
+| **Multi-language analyzers** | `Scanner.Analyzers.Lang.*` | Complete - 10 languages |
+
+### Needs Implementation
+
+| Component | Gap | Priority |
+|-----------|-----|----------|
+| **Runtime Capture Adapters** | Models exist, no Tetragon/ETW/dtrace adapters | P0 |
+| **CVE-to-Sink Orchestrator** | Schema exists, no service to trigger analysis | P0 |
+| **VEX Verdict Emitter** | Reachability results don't flow to VEX | P0 |
+| **Binary Patch Diff** | No B2R2-based patch verification | P1 |
+| **Evidence Job Queue** | Manual triggering, no automated pipeline | P1 |
+
+### Architecture Truth Table
+
+```
+Verdict Decision (ReachabilityStackEvaluator):
+
+| L1 (Static) | L2 (Binary) | L3 (Runtime) | Verdict             |
+|-------------|-------------|--------------|---------------------|
+| Reachable   | Resolved    | Not Gated    | Exploitable         |
+| Reachable   | Resolved    | Unknown      | LikelyExploitable   |
+| Reachable   | Resolved    | Gated        | Unreachable         |
+| Reachable   | Unknown     | Unknown      | PossiblyExploitable |
+| Reachable   | Not Resolved| *            | Unreachable         |
+| Not Reach   | *           | *            | Unreachable         |
+| Unknown     | *           | *            | Unknown             |
+```
+
+---
+
+## Sprint Plan
+
+### Phase 0: Foundation Validation (1 Sprint - 2 weeks)
+
+**Goal**: Validate existing components work end-to-end with manual triggering.
+
+#### Sprint S0.1: Smoke Test Existing Pipeline
+
+| Task ID | Task | Owner | Effort |
+|---------|------|-------|--------|
+| S0.1.1 | Write integration test: DotNetCallGraphExtractor -> ReachabilityAnalyzer -> ReachabilityResult | Backend | 2d |
+| S0.1.2 | Write integration test: ReachabilityStackEvaluator with mock L1/L2/L3 | Backend | 1d |
+| S0.1.3 | Write integration test: ReachabilityWitnessDsseBuilder -> signed envelope | Backend | 1d |
+| S0.1.4 | Validate CVE-symbol mapping schema with real CVE data (Log4Shell, Spring4Shell) | Backend | 1d |
+| S0.1.5 | Document gaps found during validation | Tech Lead | 1d |
+
+**Exit Criteria**:
+- All integration tests pass
+- Gap analysis document produced
+- Existing components confirmed working
+
+---
+
+### Phase 1: CVE-to-Verdict Orchestration (2 Sprints - 4 weeks)
+
+**Goal**: Enable "given CVE + image, produce reachability verdict" flow.
+
+#### Sprint S1.1: CVE-Symbol Mapping Service
+
+| Task ID | Task | Owner | Effort |
+|---------|------|-------|--------|
+| S1.1.1 | Create `ICveSymbolMappingService` interface | Backend | 0.5d |
+| S1.1.2 | Implement `PostgresCveSymbolMappingRepository` using existing schema | Backend | 2d |
+| S1.1.3 | Create `CveSymbolMappingLoader` - import from OSV/NVD advisories | Backend | 3d |
+| S1.1.4 | Create `PatchAnalysisExtractor` - parse git diffs for symbols | Backend | 2d |
+| S1.1.5 | Wire to Concelier - enrich CVE data with sink mappings | Backend | 2d |
+
+**Schema** (exists in `reachability.cve_symbol_mappings`):
+```sql
+-- Already implemented, no changes needed
+```
+
+**Interface**:
+```csharp
+public interface ICveSymbolMappingService
+{
+    Task<IReadOnlyList<VulnerableSymbol>> GetSinksForCveAsync(
+        string cveId,
+        string purl,
+        CancellationToken ct);
+
+    Task<bool> HasMappingAsync(string cveId, CancellationToken ct);
+}
+```
+
+#### Sprint S1.2: Reachability Evidence Job
+
+| Task ID | Task | Owner | Effort |
+|---------|------|-------|--------|
+| S1.2.1 | Create `ReachabilityEvidenceJob` record for queue | Backend | 1d |
+| S1.2.2 | Create `ReachabilityEvidenceJobExecutor` in Scanner.Worker | Backend | 3d |
+| S1.2.3 | Wire to existing `CallGraphSnapshot` -> `ReachabilityAnalyzer` | Backend | 2d |
+| S1.2.4 | Emit `ReachabilityStack` with L1 analysis | Backend | 1d |
+| S1.2.5 | Store result in `EvidenceDbContext` | Backend | 1d |
+| S1.2.6 | Add WebService endpoint: `POST /api/reachability/analyze` | Backend | 1d |
+
+**Job Flow**:
+```
+Request: { imageDigest, cveId, purl }
+    -> Lookup sinks from CveSymbolMappingService
+    -> Get CallGraphSnapshot from cache/compute
+    -> Run ReachabilityAnalyzer with sinks
+    -> Build ReachabilityStack (L1 only initially)
+    -> Store EvidenceBundle
+    -> Return ReachabilityResult
+```
+
+---
+
+### Phase 2: VEX Integration (2 Sprints - 4 weeks)
+
+**Goal**: Reachability results automatically influence VEX status.
+
+#### Sprint S2.1: Verdict-to-VEX Bridge
+
+| Task ID | Task | Owner | Effort |
+|---------|------|-------|--------|
+| S2.1.1 | Create `IVexStatusDeterminer` interface | Backend | 0.5d |
+| S2.1.2 | Implement verdict -> VEX status mapping | Backend | 2d |
+| S2.1.3 | Create `ReachabilityVexJustificationBuilder` | Backend | 2d |
+| S2.1.4 | Wire to VexHub - emit VEX with evidence references | Backend | 3d |
+| S2.1.5 | Add evidence URI to VEX justification | Backend | 1d |
+
+**Mapping Logic**:
+```csharp
+public VexStatus MapVerdictToVexStatus(ReachabilityVerdict verdict) => verdict switch
+{
+    ReachabilityVerdict.Exploitable => VexStatus.Affected,
+    ReachabilityVerdict.LikelyExploitable => VexStatus.Affected,
+    ReachabilityVerdict.PossiblyExploitable => VexStatus.UnderInvestigation,
+    ReachabilityVerdict.Unreachable => VexStatus.NotAffected,
+    ReachabilityVerdict.Unknown => VexStatus.UnderInvestigation,
+    _ => VexStatus.UnderInvestigation
+};
+```
+
+#### Sprint S2.2: Automated VEX Refresh
+
+| Task ID | Task | Owner | Effort |
+|---------|------|-------|--------|
+| S2.2.1 | Create `VexRefreshTrigger` - on new CVE or new scan | Backend | 2d |
+| S2.2.2 | Implement incremental VEX update (don't regenerate all) | Backend | 3d |
+| S2.2.3 | Add `vex_evidence_links` table for evidence->VEX tracking | Backend | 1d |
+| S2.2.4 | Create VexLens query: "show VEX decisions with evidence" | Backend | 2d |
+| S2.2.5 | Add UI endpoint for evidence drill-down | Frontend | 2d |
+
+---
+
+### Phase 3: Runtime Observation (2 Sprints - 4 weeks)
+
+**Goal**: Implement Layer 3 (Runtime Gating) with actual runtime data.
+
+#### Sprint S3.1: Runtime Capture Infrastructure
+
+| Task ID | Task | Owner | Effort |
+|---------|------|-------|--------|
+| S3.1.1 | Create `IRuntimeCaptureAdapter` interface | Backend | 1d |
+| S3.1.2 | Implement `TetragonAdapter` for Linux/Kubernetes | Backend | 5d |
+| S3.1.3 | Implement `EtwAdapter` for Windows | Backend | 3d |
+| S3.1.4 | Create `RuntimeEvidenceCollector` service | Backend | 2d |
+| S3.1.5 | Wire to existing `RuntimeEvidence` models | Backend | 1d |
+
+**Interface** (leverages existing models):
+```csharp
+public interface IRuntimeCaptureAdapter
+{
+    Task<RuntimeCaptureSession> StartSessionAsync(
+        RuntimeCaptureOptions options,
+        CancellationToken ct);
+
+    Task StopSessionAsync(string sessionId, CancellationToken ct);
+
+    IAsyncEnumerable<RuntimeLoadEvent> StreamEventsAsync(
+        string sessionId,
+        CancellationToken ct);
+}
+
+// Uses existing: RuntimeCaptureSession, RuntimeLoadEvent, RuntimeEvidence
+// from StellaOps.Scanner.Analyzers.Native.RuntimeCapture
+```
+
+#### Sprint S3.2: Runtime-to-Stack Integration
+
+| Task ID | Task | Owner | Effort |
+|---------|------|-------|--------|
+| S3.2.1 | Create `RuntimeEvidenceCorrelator` - map events to symbols | Backend | 3d |
+| S3.2.2 | Implement Layer 3 population from runtime evidence | Backend | 2d |
+| S3.2.3 | Update `ReachabilityStackEvaluator` to use real L3 data | Backend | 2d |
+| S3.2.4 | Add runtime evidence to DSSE attestation | Backend | 1d |
+| S3.2.5 | Create `RuntimeObservationEvidence` bundle type | Backend | 1d |
+
+**Tetragon Policy** (example for container runtime):
+```yaml
+apiVersion: cilium.io/v1alpha1
+kind: TracingPolicy
+metadata:
+  name: stella-vuln-tracing
+spec:
+  kprobes:
+    - call: "security_file_open"
+      selectors:
+        - matchActions:
+            - action: Sigkill # or just observe
+      args:
+        - index: 0
+          type: "file"
+```
+
+---
+
+### Phase 4: Binary Patch Verification (2 Sprints - 4 weeks)
+
+**Goal**: Implement patch verification for "distro says fixed" cases.
+
+#### Sprint S4.1: B2R2 Patch Diff Engine
+
+| Task ID | Task | Owner | Effort |
+|---------|------|-------|--------|
+| S4.1.1 | Create `IBinaryDiffService` interface | Backend | 1d |
+| S4.1.2 | Implement `B2R2BinaryDiffService` using existing B2R2 deps | Backend | 5d |
+| S4.1.3 | Create function similarity matching (basic) | Backend | 3d |
+| S4.1.4 | Create `PatchDiffEvidence` model | Backend | 1d |
+| S4.1.5 | Add to evidence bundle | Backend | 1d |
+
+**Note**: B2R2 is already in dependencies for binary lifting. No Ghidra needed.
+
+```csharp
+public interface IBinaryDiffService
+{
+    Task<PatchDiffResult> DiffAsync(
+        Stream vulnerableBinary,
+        Stream patchedBinary,
+        IReadOnlyList<string> targetSymbols,
+        CancellationToken ct);
+}
+
+public record PatchDiffResult(
+    bool IsPatched,
+    IReadOnlyList<FunctionDiff> ChangedFunctions,
+    double SimilarityScore,
+    string DiffSummary);
+```
+
+#### Sprint S4.2: Patch Verification Pipeline
+
+| Task ID | Task | Owner | Effort |
+|---------|------|-------|--------|
+| S4.2.1 | Create `PatchVerificationJob` for queue | Backend | 1d |
+| S4.2.2 | Implement binary fetching from registry/distro | Backend | 3d |
+| S4.2.3 | Wire to Layer 2 (Binary Resolution) | Backend | 2d |
+| S4.2.4 | Add patch verification to verdict logic | Backend | 2d |
+| S4.2.5 | Create "backport detected" VEX justification | Backend | 1d |
+
+---
+
+### Phase 5: DSSE Attestation & Policy Gate (1 Sprint - 2 weeks)
+
+**Goal**: All evidence signed, policy gates enforce requirements.
+
+#### Sprint S5.1: Attestation Pipeline
+
+| Task ID | Task | Owner | Effort |
+|---------|------|-------|--------|
+| S5.1.1 | Create `ReachabilityAttestationPublisher` | Backend | 2d |
+| S5.1.2 | Wire to Authority for real signing (replace deterministic fallback) | Backend | 2d |
+| S5.1.3 | Create `PolicyGateEvaluator` using attestations | Backend | 3d |
+| S5.1.4 | Add Rekor-compatible logging (optional) | Backend | 2d |
+| S5.1.5 | Create attestation verification endpoint | Backend | 1d |
+
+**Policy Example**:
+```yaml
+# Release gate policy
+gates:
+  - name: reachability-evidence
+    require:
+      - predicateType: "https://stella.ops/reachabilityWitness/v1"
+        conditions:
+          - verdict: ["Unreachable", "Unknown"]  # Block Exploitable
+          - signed: true
+```
+
+---
+
+### Phase 6: UI & Observability (1 Sprint - 2 weeks)
+
+**Goal**: Make evidence visible and actionable.
+
+#### Sprint S6.1: Evidence UI
+
+| Task ID | Task | Owner | Effort |
+|---------|------|-------|--------|
+| S6.1.1 | Add "Evidence" tab to finding detail view | Frontend | 3d |
+| S6.1.2 | Visualize call path (entry -> sink) | Frontend | 2d |
+| S6.1.3 | Show runtime observation timeline | Frontend | 2d |
+| S6.1.4 | Display patch diff summary | Frontend | 1d |
+| S6.1.5 | Add DSSE signature verification badge | Frontend | 1d |
+
+---
+
+## Database Migrations Required
+
+### New Tables
+
+```sql
+-- Runtime observation storage (extends existing RuntimeEvidence model)
+CREATE TABLE IF NOT EXISTS reachability.runtime_observations (
+    observation_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    scan_id UUID NOT NULL,
+    image_digest TEXT NOT NULL,
+    session_id TEXT NOT NULL,
+
+    -- Observation data
+    symbol_name TEXT,
+    observed_at TIMESTAMPTZ NOT NULL,
+    load_type TEXT,
+    process_id INTEGER,
+
+    -- Correlation
+    correlated_cve_id TEXT,
+    correlated_finding_id UUID,
+
+    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+CREATE INDEX idx_runtime_obs_image ON reachability.runtime_observations(image_digest);
+CREATE INDEX idx_runtime_obs_symbol ON reachability.runtime_observations(symbol_name);
+
+-- VEX-Evidence linkage
+CREATE TABLE IF NOT EXISTS reachability.vex_evidence_links (
+    link_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    vex_document_id UUID NOT NULL,
+    evidence_bundle_id UUID NOT NULL,
+    evidence_type TEXT NOT NULL,
+    linked_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+CREATE INDEX idx_vex_evidence_vex ON reachability.vex_evidence_links(vex_document_id);
+CREATE INDEX idx_vex_evidence_bundle ON reachability.vex_evidence_links(evidence_bundle_id);
+```
+
+---
+
+## Key Interfaces Summary
+
+### Core Services to Implement
+
+```csharp
+// CVE-to-Sink mapping
+public interface ICveSymbolMappingService
+{
+    Task<IReadOnlyList<VulnerableSymbol>> GetSinksForCveAsync(string cveId, string purl, CancellationToken ct);
+}
+
+// Runtime capture
+public interface IRuntimeCaptureAdapter
+{
+    Task<RuntimeCaptureSession> StartSessionAsync(RuntimeCaptureOptions options, CancellationToken ct);
+    IAsyncEnumerable<RuntimeLoadEvent> StreamEventsAsync(string sessionId, CancellationToken ct);
+}
+
+// Binary diff
+public interface IBinaryDiffService
+{
+    Task<PatchDiffResult> DiffAsync(Stream vulnerable, Stream patched, IReadOnlyList<string> symbols, CancellationToken ct);
+}
+
+// VEX integration
+public interface IVexStatusDeterminer
+{
+    VexStatus DetermineStatus(ReachabilityVerdict verdict);
+    VexJustification BuildJustification(ReachabilityStack stack, IReadOnlyList<string> evidenceUris);
+}
+
+// Evidence job
+public interface IReachabilityEvidenceJobExecutor
+{
+    Task<ReachabilityStack> ExecuteAsync(ReachabilityEvidenceJob job, CancellationToken ct);
+}
+```
+
+---
+
+## Risk Assessment
+
+| Risk | Mitigation | Owner |
+|------|------------|-------|
+| Tetragon requires privileged container | Provide fallback to log-based observation | Platform |
+| Binary diff performance on large binaries | Queue-based processing with timeouts | Backend |
+| CVE-symbol mapping accuracy | Confidence scores, manual curation workflow | Security |
+| Runtime observation overhead | Sampling, targeted policies | Platform |
+
+---
+
+## Success Metrics
+
+| Metric | Target | Measurement |
+|--------|--------|-------------|
+| Reachability evidence coverage | 80% of high/critical CVEs | Evidence bundle count |
+| Verdict accuracy (vs manual triage) | 90% | Audit sample |
+| VEX auto-population rate | 60% of findings | VEX with evidence links |
+| Runtime observation latency | < 5s to verdict | P95 latency |
+
+---
+
+## Timeline Summary
+
+| Phase | Sprints | Duration | Focus |
+|-------|---------|----------|-------|
+| Phase 0 | 1 | 2 weeks | Validation |
+| Phase 1 | 2 | 4 weeks | CVE-to-Verdict |
+| Phase 2 | 2 | 4 weeks | VEX Integration |
+| Phase 3 | 2 | 4 weeks | Runtime |
+| Phase 4 | 2 | 4 weeks | Binary Diff |
+| Phase 5 | 1 | 2 weeks | Attestation |
+| Phase 6 | 1 | 2 weeks | UI |
+| **Total** | **11** | **22 weeks** | |
+
+---
+
+## Appendix: Existing Code References
+
+### Reachability Stack (3-Layer Model)
+- `Scanner.Reachability/Stack/ReachabilityStack.cs` - Stack model
+- `Scanner.Reachability/Stack/ReachabilityStackEvaluator.cs` - Verdict logic
+- `Scanner.Reachability/Stack/ReachabilityLayer1.cs` - Static analysis layer
+- `Scanner.Reachability/Stack/ReachabilityLayer2.cs` - Binary resolution layer
+- `Scanner.Reachability/Stack/ReachabilityLayer3.cs` - Runtime gating layer
+
+### Evidence Models
+- `Evidence.Bundle/ReachabilityEvidence.cs` - Reachability proof
+- `Evidence.Bundle/EvidenceBundle.cs` - Container
+- `Evidence.Bundle/CallStackEvidence.cs` - Call stack trace
+- `Evidence.Bundle/DiffEvidence.cs` - Diff proof
+
+### Attestation
+- `Scanner.Reachability/Attestation/ReachabilityWitnessDsseBuilder.cs` - DSSE builder
+- `Scanner.Reachability/Attestation/ReachabilityWitnessStatement.cs` - Statement model
+- `Scanner.Reachability/Attestation/ReachabilityWitnessPublisher.cs` - Publisher
+
+### Call Graph
+- `Scanner.CallGraph/Analysis/ReachabilityAnalyzer.cs` - BFS analysis
+- `Scanner.CallGraph/Extraction/DotNet/DotNetCallGraphExtractor.cs` - .NET extraction
+
+### Runtime (Models Only)
+- `Scanner.Analyzers.Native/RuntimeCapture/RuntimeEvidence.cs` - Models
+- `Scanner.Analyzers.Native/RuntimeCapture/RuntimeCaptureOptions.cs` - Options
+
+### Lattice
+- `Scanner.Emit/Reachability/ReachabilityLattice.cs` - Score-based merge