fixes save

docs/product-advisories/27-Dec-2025 - Advisory Lens Gap Analysis and Implementation Plan.md

@@ -0,0 +1,524 @@
+# Advisory Lens - Gap Analysis and Implementation Plan
+
+**Date:** 2025-12-27
+**Status:** Under Review
+**Related Advisory:** Advisory Lens Vision Document
+
+---
+
+## Executive Summary
+
+The "Advisory Lens" vision proposes a contextual copilot that learns from organizational data (SBOM changes, reachability graphs, triage outcomes, policy decisions) to surface explainable suggestions. After comprehensive analysis against the StellaOps codebase, this advisory represents a **high-value, strategically aligned enhancement** that leverages substantial existing infrastructure while filling critical gaps.
+
+### Strategic Fit Score: 9/10
+
+**Why this matters for StellaOps:**
+- Directly amplifies the platform's core differentiator: **explainable, evidence-backed decisioning**
+- Builds on existing investments in reachability, attestations, and policy infrastructure
+- Creates defensible moat through institutional memory and deterministic replay
+- Aligns with offline-first, determinism-first architectural principles
+
+---
+
+## Gap Analysis: What Exists vs. What's Needed
+
+### 1. Signals & Learning Sources
+
+| Advisory Requirement | Existing Capability | Gap Level |
+|---------------------|---------------------|-----------|
+| **Reachability graphs** | Scanner: SmartDiff, ReachabilityDrift, 3-bit ReachabilityGate, CallGraph extractors (5 languages) | **LOW** - Already rich |
+| **SBOM deltas** | Scanner: diff-aware rescans, SmartDiffPredicate; SbomService: lineage ledger, LNM schema | **LOW** - Needs delta extraction API |
+| **VEX & triage history** | Excititor: VexCandidateEmitter, emission triggers; Findings Ledger: immutable audit trail | **MEDIUM** - Need outcome correlation |
+| **Runtime hints** | Signals: 5-factor Unknowns scoring, HOT/WARM/COLD bands; Scanner: eBPF/ETW runtime traces (Sprint 3840) | **MEDIUM** - Feature flag detection missing |
+| **Policy outcomes** | Policy: K4 lattice logic, 7-status PolicyVerdict, PolicyExplanation, SuppressionRuleEvaluator | **LOW** - Outcomes tracked |
+
+### 2. Core Loop Components
+
+| Advisory Requirement | Existing Capability | Gap Level |
+|---------------------|---------------------|-----------|
+| **Ingest & normalize** | CycloneDX/SPDX fully supported; VEX ingestion; reachability edges via CallGraph | **LOW** |
+| **Match similar situations** | **BinaryIndex.Fingerprints** exists for binary matching; **NO semantic case matching** | **HIGH** - Core gap |
+| **Rank next actions** | Signals: Unknowns scoring with decay; Policy: risk scoring | **MEDIUM** - Need action ranking |
+| **Explain with evidence** | Attestor: ProofBundle, ReasoningPredicate, ProofSpine; StellaVerdict consolidation underway | **LOW** - Strong foundation |
+| **Capture feedback** | Findings Ledger: immutable audit; VEX approval workflow | **MEDIUM** - Need feedback loop |
+
+### 3. Data Model & Storage
+
+| Advisory Requirement | Existing Capability | Gap Level |
+|---------------------|---------------------|-----------|
+| **EvidenceCase** | Attestor: EvidencePredicate, content-addressed IDs (RFC 8785) | **MEDIUM** - Need advisory-specific schema |
+| **Outcome** | PolicyVerdict, VexCandidate with proof_refs | **MEDIUM** - Need outcome consolidation |
+| **Pattern** (graph-embedding + rules) | Graph module: in-memory, needs persistent backing; BinaryIndex: fingerprints | **HIGH** - Core gap |
+| **Signed & replayable** | Attestor: DSSE, Rekor, offline verification; Replay module exists | **LOW** |
+
+### 4. Attestation Infrastructure
+
+| Advisory Requirement | Existing Capability | Gap Level |
+|---------------------|---------------------|-----------|
+| **advisory.attestation type** | Attestor supports 6+ predicate types; adding new types is documented pattern | **LOW** - Add new predicate |
+| **OCI-attached attestation** | Scanner Sprint 3850: OCI artifact storage for slices | **LOW** - Reuse pattern |
+
+### 5. UI Components
+
+| Advisory Requirement | Existing Capability | Gap Level |
+|---------------------|---------------------|-----------|
+| **Lens panel** | Angular 17 frontend exists; no "Lens" component yet | **MEDIUM** - New component |
+| **Inline hints** | VEX emission surfaces candidates in triage UI | **MEDIUM** - Extend pattern |
+| **Playbooks drawer** | Policy templates exist; no dry-run UI | **HIGH** - New feature |
+| **Evidence chips** | Attestor proof chain visualization exists | **LOW** - Reuse |
+
+---
+
+## Detailed Gap Assessment
+
+### GAP-1: Semantic Case Matching (HIGH)
+
+**What's missing:** The ability to fingerprint a situation (vuln + reachability path + context) and find similar historical cases.
+
+**What exists:**
+- `BinaryIndex.Fingerprints` for binary identity extraction
+- `Scheduler.FailureSignatureIndexer` for failure pattern indexing
+- Graph module with diff/overlay capabilities
+
+**Required:**
+- Graph embedding/fingerprint library for vulnerability situations
+- Similarity index (top-k nearest neighbor search)
+- Pattern storage with policy/outcome linkage
+
+### GAP-2: Action Ranking Engine (MEDIUM)
+
+**What's missing:** Greedy risk-per-change ranking algorithm.
+
+**What exists:**
+- Signals: Unknowns 5-factor scoring with configurable weights
+- Policy: Risk scoring via `StellaOps.Policy.Scoring`
+- SmartDiff: reachability-weighted findings
+
+**Required:**
+- Upgrade ranking algorithm (actions that remove most reachable CVEs per change)
+- Integration with SBOM delta to compute "change units"
+
+### GAP-3: Feedback Loop Integration (MEDIUM)
+
+**What's missing:** Capturing accept/modify/ignore actions to train suggestions.
+
+**What exists:**
+- Findings Ledger: immutable audit trail
+- VEX approval workflow in Excititor
+
+**Required:**
+- Feedback event schema
+- Outcome correlation service
+- Precision@k tracking
+
+### GAP-4: Playbook/Dry-Run Infrastructure (HIGH)
+
+**What's missing:** One-click policy application with preview.
+
+**What exists:**
+- Policy simulation (Scheduler: `PolicyBatchSimulationWorker`)
+- Suppression rules with override providers
+
+**Required:**
+- Dry-run API with diff preview
+- Rollback plan generation
+- Playbook templating system
+
+### GAP-5: Advisory Service (NEW MODULE)
+
+**What's missing:** Central service to compute and surface suggestions.
+
+**What exists:**
+- AdvisoryAI module (AI-assisted analysis with LLM guardrails) - can be extended
+- Scanner.WebService adjacent pattern
+
+**Required:**
+- Advisory suggestion computation service
+- REST API for suggestions
+- Background worker for proactive analysis
+
+---
+
+## Risk Assessment
+
+| Risk | Likelihood | Impact | Mitigation |
+|------|------------|--------|------------|
+| Similarity matching produces poor results | Medium | High | Start with simple heuristics; add ML gradually |
+| Performance overhead on suggestion computation | Medium | Medium | Background computation; aggressive caching |
+| User distrust of "AI suggestions" | Low | High | Always show evidence; never hide reasoning |
+| Scope creep into full ML platform | High | Medium | Phase boundaries; v1 heuristics-only |
+| Integration complexity across modules | Medium | Medium | Consolidate into single AdvisoryLens module |
+
+---
+
+## Recommendation: PROCEED with Phased Implementation
+
+### Why Proceed:
+1. **Strategic Moat:** Institutional memory is defensible
+2. **Leverage Existing:** 70%+ infrastructure already built
+3. **User Delight:** Reduces triage time measurably
+4. **Determinism Aligned:** Replay-safe suggestions fit StellaOps philosophy
+
+### Critical Success Factors:
+1. Every suggestion MUST cite prior evidence
+2. Deterministic replay of suggestion computation
+3. No opaque ML - start with interpretable heuristics
+4. Offline-first: works in air-gapped deployments
+
+---
+
+## Sprint/Task Breakdown
+
+### Phase 1: Foundation (Sprints 4000-4020)
+
+#### SPRINT_4000_0001_0001_LB_advisory_lens_core
+
+**Objective:** Create core AdvisoryLens library with data models and interfaces.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 1.1 | TODO | Define `AdvisoryCase` model (sbom_hash_from/to, vuln_id, reachable_path_hash, context_keys) |
+| 1.2 | TODO | Define `AdvisoryOutcome` model (action, reason_code, proof_refs, feedback_status) |
+| 1.3 | TODO | Define `AdvisoryPattern` model (fingerprint, rules_digest, linked_outcomes) |
+| 1.4 | TODO | Define `AdvisorySuggestion` model (action, confidence, evidence_refs, explanation) |
+| 1.5 | TODO | Create `IAdvisoryLensService` interface |
+| 1.6 | TODO | Add canonical JSON serialization with RFC 8785 |
+| 1.7 | TODO | Add content-addressed ID generation |
+
+**Files:**
+- `src/__Libraries/StellaOps.AdvisoryLens/Models/AdvisoryCase.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Models/AdvisoryOutcome.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Models/AdvisoryPattern.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Models/AdvisorySuggestion.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Services/IAdvisoryLensService.cs`
+
+#### SPRINT_4000_0001_0002_LB_graph_fingerprint
+
+**Objective:** Deterministic graph fingerprinting for reachability subgraphs.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 2.1 | TODO | Design fingerprint schema (vuln + entrypoint + path + context) |
+| 2.2 | TODO | Implement `ReachabilityFingerprintBuilder` with deterministic hashing |
+| 2.3 | TODO | Add context extraction (feature flags, env vars, policy bindings) |
+| 2.4 | TODO | Create `IGraphFingerprintService` interface |
+| 2.5 | TODO | Add serialization to BLAKE3 content-addressed ID |
+| 2.6 | TODO | Write determinism tests (same inputs = same fingerprint) |
+
+**Files:**
+- `src/__Libraries/StellaOps.AdvisoryLens/Fingerprinting/ReachabilityFingerprintBuilder.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Fingerprinting/IGraphFingerprintService.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Fingerprinting/ContextExtractor.cs`
+
+#### SPRINT_4000_0001_0003_BE_similarity_index
+
+**Objective:** Pattern similarity index with top-k retrieval.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 3.1 | TODO | Design PostgreSQL schema for patterns with GIN indexes |
+| 3.2 | TODO | Implement `PatternRepository` with similarity search |
+| 3.3 | TODO | Add Valkey cache layer for hot patterns |
+| 3.4 | TODO | Create `ISimilarityIndexService` interface |
+| 3.5 | TODO | Implement simple Jaccard similarity for v1 |
+| 3.6 | TODO | Add threshold-based noise gating |
+| 3.7 | TODO | Write integration tests with Testcontainers |
+
+**Files:**
+- `src/__Libraries/StellaOps.AdvisoryLens.Persistence/Postgres/PatternRepository.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Services/SimilarityIndexService.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Services/ISimilarityIndexService.cs`
+
+#### SPRINT_4000_0002_0001_BE_sbom_delta_service
+
+**Objective:** Extract and expose SBOM deltas for suggestion computation.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 4.1 | TODO | Create `SbomDeltaExtractor` using existing SmartDiff infrastructure |
+| 4.2 | TODO | Define delta schema (added, removed, upgraded, downgraded packages) |
+| 4.3 | TODO | Add transitive dependency tracking |
+| 4.4 | TODO | Expose `GET /api/v1/sbom/{id}/delta?to={id}` endpoint |
+| 4.5 | TODO | Add deterministic ordering to delta output |
+
+**Files:**
+- `src/SbomService/StellaOps.SbomService/Services/SbomDeltaExtractor.cs`
+- `src/SbomService/StellaOps.SbomService/Endpoints/SbomDeltaEndpoints.cs`
+
+### Phase 1 Continued: Heuristics Engine (Sprints 4010-4020)
+
+#### SPRINT_4010_0001_0001_BE_suggestion_engine
+
+**Objective:** Core suggestion computation with initial heuristics.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 5.1 | TODO | Implement `GreedyRiskPerChangeRanker` |
+| 5.2 | TODO | Implement `SubgraphSimilarityMatcher` |
+| 5.3 | TODO | Implement `NoiseGateFilter` (weak evidence threshold) |
+| 5.4 | TODO | Create `SuggestionEngine` orchestrator |
+| 5.5 | TODO | Add explanation generator with evidence links |
+| 5.6 | TODO | Configure heuristic weights via IOptions |
+
+**Files:**
+- `src/__Libraries/StellaOps.AdvisoryLens/Heuristics/GreedyRiskPerChangeRanker.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Heuristics/SubgraphSimilarityMatcher.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Heuristics/NoiseGateFilter.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Services/SuggestionEngine.cs`
+
+#### SPRINT_4010_0001_0002_BE_outcome_tracker
+
+**Objective:** Capture and correlate outcomes from policy decisions.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 6.1 | TODO | Create `OutcomeCorrelationService` |
+| 6.2 | TODO | Integrate with Findings Ledger events |
+| 6.3 | TODO | Integrate with VEX approval workflow |
+| 6.4 | TODO | Add feedback event schema |
+| 6.5 | TODO | Store outcomes with pattern linkage |
+| 6.6 | TODO | Implement precision@k tracking |
+
+**Files:**
+- `src/__Libraries/StellaOps.AdvisoryLens/Services/OutcomeCorrelationService.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Events/FeedbackEvent.cs`
+
+#### SPRINT_4010_0002_0001_BE_advisory_attestation
+
+**Objective:** New attestation type for advisory suggestions.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 7.1 | TODO | Define `AdvisoryPredicate` following existing patterns |
+| 7.2 | TODO | Add predicate type: `application/vnd.stellaops.advisory+json` |
+| 7.3 | TODO | Implement `AdvisoryAttestationBuilder` |
+| 7.4 | TODO | Add DSSE signing integration |
+| 7.5 | TODO | Create schema: `docs/schemas/stellaops-advisory.v1.schema.json` |
+| 7.6 | TODO | Add to Attestor predicate registry |
+
+**Files:**
+- `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Predicates/AdvisoryPredicate.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Attestation/AdvisoryAttestationBuilder.cs`
+- `docs/schemas/stellaops-advisory.v1.schema.json`
+
+### Phase 1: API & Integration (Sprint 4020)
+
+#### SPRINT_4020_0001_0001_BE_advisory_api
+
+**Objective:** REST API for advisory suggestions.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 8.1 | TODO | Create `AdvisoryLensController` |
+| 8.2 | TODO | Implement `GET /api/v1/advisory/suggestions?artifact={id}` |
+| 8.3 | TODO | Implement `GET /api/v1/advisory/suggestions/{id}/evidence` |
+| 8.4 | TODO | Implement `POST /api/v1/advisory/feedback` |
+| 8.5 | TODO | Add tenant isolation via RLS |
+| 8.6 | TODO | Add rate limiting and caching |
+
+**Files:**
+- `src/AdvisoryAI/StellaOps.AdvisoryAI.WebService/Controllers/AdvisoryLensController.cs`
+
+#### SPRINT_4020_0001_0002_BE_background_worker
+
+**Objective:** Background suggestion computation on SBOM/VEX changes.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 9.1 | TODO | Create `AdvisorySuggestionWorker` |
+| 9.2 | TODO | Subscribe to SBOM ingestion events |
+| 9.3 | TODO | Subscribe to VEX change events |
+| 9.4 | TODO | Implement batch suggestion computation |
+| 9.5 | TODO | Add metrics: suggestion latency, cache hit ratio |
+
+**Files:**
+- `src/AdvisoryAI/StellaOps.AdvisoryAI.Worker/Workers/AdvisorySuggestionWorker.cs`
+
+### Phase 2: UI Integration (Sprints 4030-4040)
+
+#### SPRINT_4030_0001_0001_FE_lens_panel
+
+**Objective:** "Top 3 Suggestions Today" panel for Timeline/Projects.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 10.1 | TODO | Create `LensPanelComponent` |
+| 10.2 | TODO | Design suggestion card with evidence chips |
+| 10.3 | TODO | Add "Apply as dry-run" button |
+| 10.4 | TODO | Integrate with Timeline view |
+| 10.5 | TODO | Add loading/empty states |
+
+**Files:**
+- `src/Web/StellaOps.Web/src/app/components/lens-panel/`
+
+#### SPRINT_4030_0001_0002_FE_inline_hints
+
+**Objective:** Inline hints on detail pages.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 11.1 | TODO | Create `InlineHintComponent` |
+| 11.2 | TODO | Add to vulnerability detail pages |
+| 11.3 | TODO | Add to SBOM component pages |
+| 11.4 | TODO | Style with non-obtrusive design |
+
+**Files:**
+- `src/Web/StellaOps.Web/src/app/components/inline-hint/`
+
+#### SPRINT_4040_0001_0001_FE_playbooks_drawer
+
+**Objective:** Playbook application with dry-run preview.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 12.1 | TODO | Create `PlaybookDrawerComponent` |
+| 12.2 | TODO | Implement dry-run diff view |
+| 12.3 | TODO | Add rollback plan display |
+| 12.4 | TODO | Integrate with policy application |
+| 12.5 | TODO | Add confirmation flow |
+
+**Files:**
+- `src/Web/StellaOps.Web/src/app/components/playbook-drawer/`
+
+#### SPRINT_4040_0001_0002_BE_dry_run_api
+
+**Objective:** Backend support for dry-run policy application.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 13.1 | TODO | Extend `PolicyBatchSimulationWorker` for dry-run |
+| 13.2 | TODO | Implement `POST /api/v1/advisory/apply?dryRun=true` |
+| 13.3 | TODO | Generate signed delta-verdict |
+| 13.4 | TODO | Generate rollback plan |
+| 13.5 | TODO | Add to attestation chain |
+
+**Files:**
+- `src/Policy/StellaOps.Policy.Engine/Services/DryRunService.cs`
+
+### Phase 2 Continued: Counterfactuals & Templates (Sprint 4050)
+
+#### SPRINT_4050_0001_0001_BE_counterfactuals
+
+**Objective:** "Had you done X, Y wouldn't have happened" analysis.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 14.1 | TODO | Design counterfactual computation model |
+| 14.2 | TODO | Implement `CounterfactualAnalyzer` |
+| 14.3 | TODO | Integrate with historical findings |
+| 14.4 | TODO | Add to suggestion explanations |
+
+**Files:**
+- `src/__Libraries/StellaOps.AdvisoryLens/Analysis/CounterfactualAnalyzer.cs`
+
+#### SPRINT_4050_0001_0002_BE_playbook_templates
+
+**Objective:** Turn accepted advisories into reusable playbooks.
+
+| Task | Status | Description |
+|------|--------|-------------|
+| 15.1 | TODO | Design playbook template schema |
+| 15.2 | TODO | Implement `PlaybookTemplateService` |
+| 15.3 | TODO | Add parameterization support |
+| 15.4 | TODO | Create template storage |
+| 15.5 | TODO | Add sharing/team-scope controls |
+
+**Files:**
+- `src/__Libraries/StellaOps.AdvisoryLens/Playbooks/PlaybookTemplate.cs`
+- `src/__Libraries/StellaOps.AdvisoryLens/Playbooks/PlaybookTemplateService.cs`
+
+---
+
+## Acceptance Criteria for v1
+
+| Metric | Target |
+|--------|--------|
+| Suggestions with prior case evidence | >= 70% |
+| Acceptance rate (accepted or edited) | >= 50% in pilot |
+| Mean triage time reduction | >= 30% on reachable CVE bursts |
+| Determinism | Same inputs = identical suggestions |
+| Offline support | Full functionality in air-gapped mode |
+
+---
+
+## Architecture Decision Records
+
+### ADR-1: Module Placement
+
+**Decision:** Create `StellaOps.AdvisoryLens` as new library under `src/__Libraries/`, extend `AdvisoryAI` module for hosting.
+
+**Rationale:**
+- AdvisoryAI already exists with AI guardrails
+- Keep core logic in reusable library
+- WebService/Worker pattern matches existing modules
+
+### ADR-2: Heuristics Before ML
+
+**Decision:** Phase 1 uses deterministic heuristics only; ML deferred to Phase 3+.
+
+**Rationale:**
+- Determinism is core StellaOps principle
+- Explainability requires interpretable rules
+- ML adds complexity without proven value
+- Easy to add ML later via strategy pattern
+
+### ADR-3: Pattern Storage
+
+**Decision:** PostgreSQL with GIN indexes + Valkey cache.
+
+**Rationale:**
+- Consistent with platform data strategy
+- Supports offline operation
+- GIN indexes efficient for similarity search
+- Valkey provides hot pattern caching
+
+### ADR-4: Attestation Type
+
+**Decision:** New predicate `application/vnd.stellaops.advisory+json`.
+
+**Rationale:**
+- Follows established Attestor predicate pattern
+- Enables signed, replayable suggestions
+- OCI attachment for portability
+
+---
+
+## Dependencies & Prerequisites
+
+| Dependency | Status | Notes |
+|------------|--------|-------|
+| StellaVerdict consolidation | In Progress | Sprint 1227.0014.0001 |
+| Scanner SmartDiff | Complete | Provides reachability basis |
+| Findings Ledger | Complete | Outcome tracking |
+| Attestor ProofChain | Complete | Evidence linking |
+| Angular 17 frontend | Complete | UI foundation |
+
+---
+
+## Related Documents
+
+- `docs/modules/advisory-ai/architecture.md` (to be created)
+- `docs/modules/scanner/reachability-drift.md`
+- `docs/modules/attestor/architecture.md`
+- `docs/modules/policy/architecture.md`
+
+---
+
+## Appendix: Module Inventory Leveraged
+
+| Module | Capabilities Used |
+|--------|------------------|
+| Scanner | SmartDiff, ReachabilityDrift, CallGraph, ReachabilityGate, VulnSurfaces |
+| Policy | K4 lattice, PolicyVerdict, SuppressionRules, RiskScoring |
+| Signals | Unknowns scoring, HOT/WARM/COLD bands, decay |
+| Attestor | DSSE, ProofChain, EvidencePredicate, ReasoningPredicate |
+| VexLens | VEX consensus |
+| Excititor | VexCandidateEmitter, emission triggers |
+| SbomService | Lineage ledger, LNM schema |
+| Graph | Query/diff/overlay APIs |
+| Findings Ledger | Immutable audit trail |
+| BinaryIndex | Fingerprinting patterns |
+
+---
+
+*This advisory was generated based on comprehensive codebase analysis. All sprint estimates are scope-based, not time-based.*