partly or unimplemented features - now implemented

This commit is contained in:
master
2026-02-09 08:53:51 +02:00
parent 1bf6bbf395
commit 4bdc298ec1
674 changed files with 90194 additions and 2271 deletions

View File

@@ -0,0 +1,48 @@
# Ground-Truth Corpus with Reachability Tiers (R0-R4)
## Module
Scanner
## Status
IMPLEMENTED
## Description
A curated corpus of small service applications ("toys") with manually-labeled reachability tiers (R0-R4) for every known vulnerability, enabling precision/recall measurement of the scanner's reachability analysis engine. Each toy service contains a known vulnerability at a specific reachability tier, with a labels.yaml defining the ground truth.
## What's Implemented
- **Golden Fixture Tests**:
- `src/Scanner/__Tests/StellaOps.Scanner.SmartDiff.Tests/` - Golden fixture tests comparing scanner output against expected results for diff-aware rescanning scenarios
- `src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/` - Reachability analysis tests with frozen call-graph fixtures and expected reachability classifications
- **Reachability Tier Infrastructure**:
- `src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Slices/ReachabilitySlice.cs` - `ReachabilitySlice` model with confidence tiers (Confirmed, Likely, Present, Unreachable) used for classification
- `src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Surfaces/SurfaceAwareReachabilityAnalyzer.cs` - `SurfaceAwareReachabilityAnalyzer` performing reachability classification with boundary extractors
- **Determinism Tests**:
- Existing test infrastructure uses frozen JSON fixtures as inputs with expected output assertions, demonstrating the pattern needed for ground-truth corpus tests
## What's Missing
- **Toy Service Corpus**: No `/toys/svc-XX-<name>/` directory structure containing small service applications with known vulnerabilities at specific reachability tiers
- **labels.yaml Schema**: No `labels.yaml` file per toy service defining ground-truth reachability tier (R0=unreachable, R1=present-in-dependency, R2=imported-not-called, R3=called-not-reachable-from-entrypoint, R4=reachable-from-entrypoint) for each CVE
- **Precision/Recall Measurement**: No test harness that runs the scanner against the corpus and computes precision, recall, and F1 score per reachability tier
- **Corpus CI Job**: No dedicated CI job that runs corpus evaluation and fails on regression (precision/recall drop below threshold)
- **Multi-Language Coverage**: Corpus should cover at least Java, Node.js, Python, Go, .NET, and Ruby to validate per-language call-graph extractors
## Implementation Plan
1. Create `/toys/` directory with initial toy services: `svc-01-log4shell-java/`, `svc-02-prototype-pollution-node/`, `svc-03-pickle-deserialization-python/`, `svc-04-text-template-go/`, `svc-05-xmlserializer-dotnet/`, `svc-06-erb-injection-ruby/`
2. For each toy service, create minimal source code with a known CVE at a specific reachability tier
3. Create `labels.yaml` per service with CVE ID, expected reachability tier (R0-R4), and human explanation
4. Create `StellaOps.Scanner.GroundTruth.Tests` project with a corpus runner that scans each toy service, extracts reachability classifications, and compares against labels.yaml
5. Implement precision/recall/F1 computation per tier and per language
6. Add CI job `scanner-ground-truth` that fails on regression below configurable thresholds
7. Document corpus contribution guidelines for adding new toy services
## E2E Test Plan
- [ ] Scan `svc-01-log4shell-java/` and verify the scanner classifies CVE-2021-44228 at the expected reachability tier from labels.yaml
- [ ] Scan all toy services and verify precision >= 90% and recall >= 85% across all reachability tiers
- [ ] Verify the corpus runner produces a summary report with per-tier and per-language precision/recall/F1 scores
- [ ] Verify adding a new toy service with labels.yaml is automatically picked up by the corpus runner without code changes
- [ ] Verify the CI job fails when a scanner change causes precision or recall to drop below the configured threshold
- [ ] Verify the corpus covers at least 3 different languages and all 5 reachability tiers (R0-R4)
## Related Documentation
- Source: See feature catalog
- Architecture: `docs/modules/scanner/architecture.md`

View File

@@ -0,0 +1,50 @@
# Idempotent Attestation Submission
## Module
Scanner
## Status
IMPLEMENTED
## Description
Ensures that attestation submissions (verdict push to OCI registry, Rekor transparency log entries) are idempotent: resubmitting the same attestation produces no duplicate entries and returns the existing entry reference. Handles transient failures with retry logic that avoids creating duplicate transparency log entries.
## What's Implemented
- **Verdict Push Stage**:
- `src/Scanner/StellaOps.Scanner.Worker/Processing/VerdictPushStageExecutor.cs` - `VerdictPushStageExecutor` scan pipeline stage that pushes signed verdict attestations to the OCI registry and triggers downstream evidence publication
- **OCI Verdict Publishing**:
- `src/Scanner/__Libraries/StellaOps.Scanner.Storage.Oci/VerdictOciPublisher.cs` - `VerdictOciPublisher` publishing signed verdict DSSE envelopes to OCI registries as attestation artifacts
- **Rekor Integration**:
- `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Rekor/EnhancedRekorProofBuilder.Build.cs` - Builds Rekor inclusion proofs for attestations
- `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Rekor/EnhancedRekorProofBuilder.Validate.cs` - Validates Rekor inclusion proofs
- `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Pipeline/RekorEntry.cs` - `RekorEntry` model for transparency log entries
- **Evidence Pipeline**:
- `src/Scanner/__Libraries/StellaOps.Scanner.Evidence/` - Evidence collection and bundling infrastructure
## What's Missing
- **Idempotency Key Generation**: No content-addressed idempotency key derived from attestation payload hash to detect duplicate submissions before sending to Rekor
- **Rekor Duplicate Detection**: No pre-submission check against Rekor search API to verify whether an attestation with the same content hash already exists in the log
- **Retry with Deduplication**: `VerdictPushStageExecutor` lacks explicit retry logic that distinguishes between "already submitted" (success) and "transient failure" (retry) responses from Rekor/OCI
- **OCI Tag Idempotency**: `VerdictOciPublisher` does not check for existing attestation tags before pushing, potentially creating duplicate OCI artifacts
- **Submission Receipt Cache**: No local cache of recently-submitted attestation hashes to short-circuit redundant submissions within the same scan session
## Implementation Plan
1. Add `AttestationIdempotencyKey` generation in `StellaOps.Scanner.Evidence` using SHA-256 of the canonical DSSE envelope payload
2. Add Rekor search-by-hash pre-check in `EnhancedRekorProofBuilder` before submitting new entries
3. Add retry policy to `VerdictPushStageExecutor` with idempotency-aware error classification (409 Conflict = success, 5xx = retry, 4xx = fail)
4. Add OCI manifest existence check in `VerdictOciPublisher` before pushing attestation artifacts
5. Add in-memory submission receipt cache with TTL for short-circuiting redundant submissions
6. Add unit tests for idempotency key generation, duplicate detection, and retry classification
## E2E Test Plan
- [ ] Submit a verdict attestation to Rekor and verify a log entry is created with correct content hash
- [ ] Resubmit the same attestation and verify no duplicate Rekor entry is created; the existing entry reference is returned
- [ ] Simulate a transient Rekor failure (503) during submission and verify the retry logic resubmits successfully without creating duplicates
- [ ] Push a verdict attestation to OCI and verify the artifact tag is created
- [ ] Re-push the same attestation to OCI and verify no duplicate artifact is created
- [ ] Verify the idempotency key is deterministically derived from the DSSE envelope payload (same payload = same key across scanner instances)
- [ ] Verify the submission receipt cache prevents redundant network calls for attestations submitted within the same scan session
## Related Documentation
- Source: See feature catalog
- Architecture: `docs/modules/scanner/architecture.md`

View File

@@ -0,0 +1,55 @@
# Stack-Trace/Exploit Path View
## Module
Scanner
## Status
IMPLEMENTED
## Description
A dedicated "Stack-Trace Lens" UX component that renders exploit paths as interactive stack-trace visualizations, allowing security engineers to trace the call chain from entrypoint to vulnerable function. Combines backend exploit path grouping with a frontend visualization component.
## What's Implemented
- **Exploit Path Grouping (Backend)**:
- `src/Scanner/__Libraries/StellaOps.Scanner.Triage/Services/IExploitPathGroupingService.cs` - `IExploitPathGroupingService` interface for grouping triage findings by their exploit path similarity
- `src/Scanner/__Libraries/StellaOps.Scanner.Triage/Models/ExploitPath.cs` - `ExploitPath` model representing a call chain from entrypoint to vulnerable function
- **Path Explanation (Backend)**:
- `src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Explanation/PathExplanationService.cs` - `PathExplanationService` generating human-readable explanations of reachability paths
- `src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Explanation/PathRenderer.cs` - `PathRenderer` rendering call-graph paths as structured output (text, JSON, diagram-ready format)
- **Triage API**:
- `src/Scanner/StellaOps.Scanner.WebService/Endpoints/Triage/TriageInboxEndpoints.cs` - `TriageInboxEndpoints` REST API serving triage findings with exploit path data to frontend consumers
- **Call Path Models**:
- `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Statements/WitnessCallPathNode.cs` - `WitnessCallPathNode` model for individual nodes in a call path
- `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Statements/WitnessPathNode.cs` - `WitnessPathNode` model for path traversal nodes
## What's Missing
- **Stack-Trace Lens UI Component**: No dedicated Angular component in `src/Web/` that renders exploit paths as interactive stack-trace visualizations with:
- Collapsible call-chain frames (entrypoint -> intermediate calls -> vulnerable function)
- Syntax-highlighted source snippets at each frame (when source mapping is available)
- Reachability tier badge per frame (Confirmed/Likely/Present)
- Click-to-expand frame details (file path, line number, function signature)
- **Path Comparison View**: No side-by-side comparison of multiple exploit paths reaching the same vulnerable function from different entrypoints
- **Path Sharing/Export**: No URL-addressable deep links to specific exploit paths for sharing in issue trackers
- **Path Filtering**: No filtering of exploit paths by reachability confidence, entrypoint type, or path length
## Implementation Plan
1. Create `ExploitPathViewComponent` in `src/Web/` as an Angular component consuming the TriageInboxEndpoints exploit path API
2. Implement collapsible stack-frame rendering with entrypoint -> call chain -> sink visualization
3. Add source snippet rendering using source-map data when available from the scanner
4. Add reachability tier badges per frame using the PathExplanationService output
5. Implement path comparison view for vulnerabilities with multiple exploit paths
6. Add deep-link URL routing for direct navigation to specific exploit paths
7. Add path filtering controls (confidence tier, entrypoint type, path length)
## E2E Test Plan
- [ ] Navigate to a triage finding with a known exploit path and verify the Stack-Trace Lens component renders the full call chain from entrypoint to vulnerable function
- [ ] Verify each frame in the stack trace shows the correct function signature, file path, and reachability tier badge
- [ ] Click to expand a frame and verify source snippet is displayed (when source mapping is available)
- [ ] Verify collapsing intermediate frames hides call chain details while keeping entrypoint and sink visible
- [ ] Open the path comparison view for a vulnerability with multiple exploit paths and verify side-by-side rendering
- [ ] Copy the deep link URL for a specific exploit path, navigate to it in a new browser tab, and verify the same path is displayed
- [ ] Apply path filtering by reachability confidence and verify only paths matching the selected tier are shown
## Related Documentation
- Source: See feature catalog
- Architecture: `docs/modules/scanner/architecture.md`

View File

@@ -0,0 +1,51 @@
# VEX Decision Filter with Reachability
## Module
Scanner
## Status
IMPLEMENTED
## Description
A dedicated reachability-aware VEX decision filter that combines VEX consensus data (from VexLens) with reachability classification to produce filtered vulnerability lists. Findings with "not_affected" VEX status and "unreachable" reachability classification are automatically suppressed, while findings with "exploitable" VEX status and "confirmed reachable" classification are elevated.
## What's Implemented
- **VEX Integration**:
- `src/Scanner/__Libraries/StellaOps.Scanner.ChangeTrace/Integration/IVexLensClient.cs` - `IVexLensClient` interface for querying VEX consensus data from the VexLens service
- `src/Scanner/__Libraries/StellaOps.Scanner.ChangeTrace/TrustDeltaCalculator.cs` - `TrustDeltaCalculator` computing trust deltas that incorporate VEX status changes
- **VEX Gate Service**:
- `src/Scanner/__Libraries/StellaOps.Scanner.Gate/VexGateService.cs` - `VexGateService` pre-triage VEX gating with Pass/Warn/Block decisions based on VEX status
- `src/Scanner/__Libraries/StellaOps.Scanner.Gate/VexGatePolicyEvaluator.cs` - `VexGatePolicyEvaluator` evaluating gate rules (block-exploitable-reachable, pass-vendor-not-affected, etc.)
- **Reachability Classification**:
- `src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Slices/ReachabilitySlice.cs` - `ReachabilitySlice` with confidence tiers (Confirmed, Likely, Present, Unreachable)
- `src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Surfaces/SurfaceAwareReachabilityAnalyzer.cs` - `SurfaceAwareReachabilityAnalyzer` performing reachability classification
- **Triage Queries**:
- `src/Scanner/StellaOps.Scanner.WebService/Endpoints/Triage/TriageInboxEndpoints.cs` - Triage API returning findings that incorporate reachability data
## What's Missing
- **Dedicated VEX+Reachability Filter Component**: No standalone `VexReachabilityDecisionFilter` that combines VEX consensus status with reachability classification into a single filtering decision (the current VexGateService evaluates VEX rules but does not deeply integrate reachability slice confidence tiers)
- **Combined Score Matrix**: No explicit decision matrix mapping (VEX status x Reachability tier) -> filter action (suppress, elevate, pass-through, flag-for-review)
- **Filter API Endpoint**: No REST endpoint that accepts a list of findings and returns the filtered list with VEX+reachability decision annotations
- **Filter Audit Trail**: No audit logging of VEX+reachability filter decisions separate from the VEX gate audit trail
- **Reachability-Weighted VEX Override**: No logic for reachability to override VEX decisions (e.g., vendor says "not_affected" but scanner detects confirmed reachable call path)
## Implementation Plan
1. Create `VexReachabilityDecisionFilter` in `StellaOps.Scanner.Gate` or a new `StellaOps.Scanner.VexFilter` library that combines `IVexLensClient` data with `ReachabilitySlice` classification
2. Define decision matrix: (not_affected + Unreachable) -> suppress, (exploitable + Confirmed) -> elevate, (not_affected + Confirmed) -> flag-for-review, etc.
3. Add `VexReachabilityFilterEndpoints` in `StellaOps.Scanner.WebService` for querying filtered finding lists
4. Add audit logging for filter decisions with VEX source, reachability tier, and resulting action
5. Implement reachability override logic: when reachability contradicts VEX status, flag the finding for human review
6. Add unit tests for all cells in the decision matrix
## E2E Test Plan
- [ ] Submit a finding with VEX "not_affected" and reachability "Unreachable" and verify the filter suppresses it
- [ ] Submit a finding with VEX "exploitable" and reachability "Confirmed" and verify the filter elevates it with highest priority
- [ ] Submit a finding with VEX "not_affected" but reachability "Confirmed" (contradiction) and verify the filter flags it for human review
- [ ] Submit a finding with no VEX data and reachability "Likely" and verify the filter passes it through with reachability-only annotation
- [ ] Query the filter API endpoint with a batch of 50 findings and verify the response includes correct filter actions and decision annotations for each
- [ ] Verify the audit trail captures each filter decision with VEX source, reachability tier, decision matrix cell, and timestamp
- [ ] Verify the filter correctly handles findings where reachability data is unavailable (falls back to VEX-only decision)
## Related Documentation
- Source: See feature catalog
- Architecture: `docs/modules/scanner/architecture.md`

View File

@@ -0,0 +1,65 @@
# Vulnerability-First Triage UX with Exploit Path Grouping and Proof Bundles
## Module
Scanner (with Attestor proof bundle integration)
## Status
IMPLEMENTED
## Description
A vulnerability-first triage inbox where findings are grouped by exploit path similarity rather than by CVE or component. Security engineers see clusters of findings that share the same attack vector (entrypoint -> call chain -> sink), enabling batch triage. Backend triage service with DB context, reachability subgraph extraction, exploit path grouping, and proof generation exist. UI triage inbox and queue components are partially complete.
## What's Implemented
- **Exploit Path Grouping (Backend)**:
- `src/Scanner/__Libraries/StellaOps.Scanner.Triage/Services/IExploitPathGroupingService.cs` -- interface grouping findings by exploit path similarity
- `src/Scanner/__Libraries/StellaOps.Scanner.Triage/Models/ExploitPath.cs` -- call chain from entrypoint to vulnerable function
- `src/Scanner/__Tests/StellaOps.Scanner.Triage.Tests/` -- unit tests for exploit path grouping logic
- **Triage Database and API**:
- `src/Scanner/__Libraries/StellaOps.Scanner.Triage/Entities/TriageDbContext.cs` -- EF Core database context
- `src/Scanner/__Libraries/StellaOps.Scanner.Triage/Entities/TriageFinding.cs` -- finding entity with reachability classification and exploit path reference
- `src/Scanner/StellaOps.Scanner.WebService/Endpoints/Triage/TriageInboxEndpoints.cs` -- REST API with exploit path grouping support
- **Triage Lanes**: `TriageEnums.cs` -- ACTIVE, BLOCKED, MUTED_REACH, MUTED_VEX
- **Triage Status Service**: `TriageStatusService.cs` -- state transitions across lanes
- **Evidence Integration**: `ProofBundleEndpoints.cs` -- evidence bundles for triage decisions
- **Proof Graph Service**: `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Graph/InMemoryProofGraphService.cs` (with `.Mutation`, `.Queries`, `.Subgraph`) -- proof graph with subgraph extraction for evidence grouping
- **Micro Witness Evidence**: `MicroWitnessFunctionEvidence.cs`, `MicroWitnessBinaryRef.cs`, `MicroWitnessCveRef.cs` -- function-level reachability evidence
- **Reachability Witness Payload**: `ReachabilityWitnessPayload.cs` (with `.Path`), `WitnessCallPathNode.cs` -- witness payload with exploit call paths
- **Proof Spine System**: `ProofSpineRequest.cs`, `ProofSpineResult.cs` -- proof spine for evidence bundle assembly
- **Sigstore/Evidence Pack Builders**: `SigstoreBundleBuilder.cs`, `ReleaseEvidencePackBuilder.cs` -- builds proof bundles
## What's Missing
- **Triage Inbox UI Component**: No Angular component implementing the vulnerability-first triage inbox with exploit path cluster view, batch triage actions, cluster expansion, sort/filter by cluster size/severity/reachability
- **Exploit Path Similarity Algorithm**: The `IExploitPathGroupingService` interface exists but the clustering/similarity algorithm completeness is unclear
- **Batch Triage API**: No REST endpoint for applying a single triage decision to all findings in an exploit path cluster
- **Cluster Statistics**: No API endpoint returning per-cluster severity and reachability distributions
- **Triage Dashboard**: No dashboard showing triage progress (clusters triaged vs. remaining, MTTR per cluster)
- **Triage Action Workflow**: No triage action buttons (accept risk, suppress, escalate, remediate) with DSSE-signed action records
- **Triage Priority Scoring**: No scoring that prioritizes vulnerabilities by exploit path depth and reachability confidence
- **Inline Proof Bundle Viewer**: No inline viewer showing proof bundle contents (DSSE envelope, Rekor receipt, Merkle proof) within the triage UI
## Implementation Plan
- Complete exploit path similarity algorithm using common call-chain prefix grouping with configurable similarity threshold
- Add `BatchTriageEndpoints` for applying triage decisions to entire exploit path clusters
- Add cluster statistics endpoint returning per-cluster severity and reachability distributions
- Create `TriageInboxComponent` in `src/Web/` with exploit path cluster view, batch actions, and filtering
- Implement triage action workflow with DSSE-signed action records
- Add triage priority scoring based on path depth and confidence
- Build inline proof bundle viewer
- Create triage dashboard with progress metrics
- Add e2e tests for grouping, triage actions, and batch operations
## E2E Test Plan
- [ ] Open triage inbox and verify findings are grouped by exploit path similarity
- [ ] Expand a cluster and verify all individual findings with CVE, severity, and reachability tier
- [ ] Apply batch "mute" action to entire cluster and verify all findings move to MUTED_REACH lane
- [ ] Sort clusters by highest severity and verify correct ordering
- [ ] Filter clusters by reachability tier "Confirmed"
- [ ] Open triage dashboard and verify correct counts
- [ ] Verify batch triage API returns correct affected finding count
## Related Documentation
- Triage infrastructure: `src/Scanner/__Libraries/StellaOps.Scanner.Triage/`
- Proof graph: `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Graph/`
## Merged From
- `attestor/vulnerability-first-triage-ux-with-exploit-path-grouping-and-proof-bundles.md` (deleted)