docs consolidation and others

docs/modules/vex-lens/guides/aggregation.md

@@ -0,0 +1,43 @@
+# VEX Observations and Linksets
+
+This document describes how StellaOps stores and correlates VEX without rewriting upstream sources.
+
+## Observations (Append-Only Evidence)
+
+An observation is the immutable record of a VEX document *as received*.
+
+Recorded fields typically include:
+
+- **Provenance:** tenant, provider/issuer identity, retrieved/received timestamps (UTC), signature verification status, trust metadata, and content digest.
+- **Raw payload:** stored losslessly to support audit and offline verification.
+- **Derived indexes:** extracted identifiers and tuples used to correlate evidence (without replacing the raw payload).
+
+If upstream republishes a VEX document, the new version is stored as a new observation with a `supersedes` relationship.
+
+## Normalization (Projection Only)
+
+Normalization produces tuples used by downstream layers:
+
+- `(vulnerabilityId, productKey)` keys for correlation
+- VEX status and justification
+- Optional scope and validity windows
+
+Normalization is a projection; it does not delete, mutate, or merge upstream evidence.
+
+## Linksets (Correlation Without Precedence)
+
+Linksets group tuples that refer to the same conceptual product-vulnerability pair.
+
+Properties:
+
+- **Deterministic:** linkset identifiers are derived from canonical, sorted key material.
+- **Conflict-preserving:** disagreements remain visible (status, scope, justification, issuer identity).
+- **Provenance-linked:** every item in a linkset references its source observation digest and issuer metadata.
+
+Downstream consumers (Policy/Console/Exports) use linksets to explain what disagrees and why.
+
+## References
+
+- `docs/VEX_CONSENSUS_GUIDE.md`
+- `docs/modules/excititor/architecture.md`
+- `docs/modules/vex-lens/architecture.md`

docs/modules/vex-lens/guides/consensus-algorithm.md

@@ -0,0 +1,37 @@
+# VEX Consensus Algorithm (Deterministic)
+
+This document describes the consensus computation at a high level. It is not an API contract; see `docs/vex/consensus-json.md` and `docs/vex/consensus-api.md` for payload and endpoint details.
+
+## Inputs
+
+- Normalized VEX tuples (status, justification, scope, timestamp, source digest)
+- Issuer trust registry (tiers, weights, verification state)
+- Optional policy precedence rules for how to treat conflicts
+
+## Grouping and Ordering
+
+1. Group tuples by correlation key (typically `(artifactId/productKey, vulnerabilityId)` per tenant).
+2. Apply a stable sort for evaluation, commonly:
+   - Most recent statement first (timestamp)
+   - Higher trust tier/weight first
+   - Higher verification confidence first
+
+Ordering must be deterministic for identical inputs.
+
+## Lattice Join
+
+Consensus uses a lattice-style join to avoid false safety:
+
+- Model states with an explicit uncertainty ordering (e.g., `unknown` and `under_investigation` remain meaningful outcomes).
+- Preserve conflicts when competing issuers disagree at comparable precedence.
+
+The output includes:
+
+- Effective status
+- Confidence/weight summary
+- References to source statements (digests)
+- Conflict list (who disagrees and how)
+
+## References
+
+- `docs/modules/vex-lens/architecture.md`

docs/modules/vex-lens/guides/consensus-api.md

@@ -0,0 +1,23 @@
+# VEX Consensus API (Contract Overview)
+
+This document provides a high-level overview of consensus endpoints. The gateway and module dossiers define the authoritative routing, auth, and schema details.
+
+## Common Headers
+
+- `Authorization: Bearer <token>` (or DPoP where configured)
+- `X-StellaOps-Tenant: <tenantId>` (required)
+
+## Typical Endpoints
+
+- `GET /v1/vex/consensus?artifact=...&advisory=...` — returns a single consensus record (plus conflicts).
+- `GET /v1/vex/conflicts` — list outstanding conflicts with summary metadata.
+- `GET /v1/vex/consensus/export` — deterministic JSONL export for Offline Kit / Export Center bundles.
+- `POST /v1/vex/trust/weights` — tenant-admin update of trust tiers/weights (when enabled).
+
+## Output Shape
+
+See `docs/vex/consensus-json.md` for the record payload reference.
+
+## References
+
+- `docs/modules/vex-lens/architecture.md`

docs/modules/vex-lens/guides/consensus-console.md

@@ -0,0 +1,23 @@
+# VEX Consensus in the Console
+
+This document describes how the Console should surface VEX evidence and consensus to keep triage explainable.
+
+## What the Console Should Show
+
+- Effective VEX status (`affected`, `not_affected`, `under_investigation`, `fixed`) with confidence/tier label.
+- Provenance for each source statement: issuer identity, signature verification state, timestamps, and digests.
+- Conflicts as first-class facts (what disagrees, at what trust tier), not hidden UI logic.
+- Snapshot identity and staleness budgets when operating against Offline Kit snapshots.
+
+## Evidence Linking
+
+Every displayed fact should link back to:
+
+- The raw VEX observation(s)
+- Linkset correlation view (what was grouped and why)
+- The policy explanation that consumed the evidence (when shown in triage)
+
+## References
+
+- Operator guide: `docs/UI_GUIDE.md`
+- VEX conceptual guide: `docs/VEX_CONSENSUS_GUIDE.md`

docs/modules/vex-lens/guides/consensus-json.md

@@ -0,0 +1,47 @@
+# VEX Consensus JSON (Payload Reference)
+
+This document describes the *shape* of consensus records returned by consensus APIs and exported in Offline Kit snapshots. Field names and structure may vary slightly by gateway serialization, but the semantic contract is stable.
+
+## Consensus Record (Conceptual)
+
+```json
+{
+  "artifact": "pkg:rpm/redhat/openssl@3.0.9",
+  "advisory": "CVE-2025-13579",
+  "status": "not_affected",
+  "confidence": 0.92,
+  "issued_at": "2025-08-30T12:05:00Z",
+  "consensus_digest": "sha256:…",
+  "derived_from": [
+    {
+      "source_digest": "sha256:…",
+      "issuer": "vendor:redhat",
+      "status": "not_affected",
+      "timestamp": "2025-08-30T12:00:00Z",
+      "trust": { "tier": "vendor", "weight": 1.0 },
+      "verification": { "signature": "verified" }
+    }
+  ],
+  "conflicts": [
+    {
+      "source_digest": "sha256:…",
+      "issuer": "vendor:upstream",
+      "status": "affected",
+      "timestamp": "2025-08-29T00:00:00Z",
+      "trust": { "tier": "vendor", "weight": 0.8 }
+    }
+  ]
+}
+```
+
+## Determinism Rules
+
+- Timestamps are UTC ISO-8601.
+- Arrays (`derived_from`, `conflicts`) are in a stable order (typically the evaluation sort order).
+- `consensus_digest` is computed over canonical, deterministic JSON bytes for the record (or over an equivalent canonical projection).
+
+## Related Docs
+
+- Algorithm overview: `docs/vex/consensus-algorithm.md`
+- Endpoints: `docs/vex/consensus-api.md`
+- Module details: `docs/modules/vex-lens/architecture.md`

docs/modules/vex-lens/guides/consensus-overview.md

@@ -0,0 +1,31 @@
+# VEX Evidence and Consensus (Detailed)
+
+This document complements `docs/VEX_CONSENSUS_GUIDE.md` with implementation-oriented detail: what objects exist, how evidence is correlated without rewriting sources, and what "consensus" means in practice.
+
+## Pipeline (Evidence First)
+
+1. **Ingest raw VEX** as immutable observations (append-only, provenance preserved).
+2. **Normalize** observations into tuples used for correlation and UI display.
+3. **Correlate** tuples into deterministic linksets (grouping without merge or precedence).
+4. **Compute consensus (optional)** using issuer trust and lattice rules to produce an “effective” status and conflict summary.
+5. **Expose evidence** to Policy Engine, Console, and Vulnerability Explorer; include in Offline Kit snapshots.
+
+## Core Objects
+
+- **Raw observation:** upstream OpenVEX/CSAF/CycloneDX payload stored losslessly with provenance (issuer/provider, receive time, signature verification, content digest).
+- **Normalized tuple:** extracted fields used for correlation and decisioning, typically `(vulnerabilityId, productKey, status, justification?, scope?, timestamp, sourceDigest)`.
+- **Linkset:** a correlation group tying multiple tuples to the same conceptual `(vulnerabilityId, productKey)` without collapsing disagreements.
+- **Consensus record:** a deterministic summary for a linkset: effective status, confidence/weight, and conflict list (still referencing raw evidence).
+
+## Determinism Guarantees
+
+- Canonical UTF-8 JSON bytes are hashed to compute stable digests for raw observations.
+- Linkset IDs are derived from canonical, sorted key material.
+- Consensus outputs are stable for identical inputs: ordering, timestamps, and digests are deterministic.
+
+## Where This Lives
+
+- Ingestion, raw store, and linksets: `docs/modules/excititor/architecture.md`
+- Consensus and issuer trust: `docs/modules/vex-lens/architecture.md`
+- Console/operator view: `docs/UI_GUIDE.md`
+- Triage model: `docs/VULNERABILITY_EXPLORER_GUIDE.md`

docs/modules/vex-lens/guides/explorer-integration.md

@@ -0,0 +1,25 @@
+# VEX Integration with Vulnerability Explorer
+
+The Vulnerability Explorer and triage surfaces treat VEX as first-class evidence: operator decisions should be explainable, replayable, and linked to provenance.
+
+## Triage View Expectations
+
+- Show effective VEX status alongside policy outcome and reachability/impact signals.
+- Make conflicts visible and navigable (issuer list, trust tiers, verification state).
+- Provide deep links from the triage view into VEX evidence (raw observations/linksets) and to policy explain traces.
+
+## Filtering and Lanes
+
+VEX evidence commonly affects:
+
+- Default lane placement (e.g., `MUTED_VEX` vs `ACTIVE`)
+- Whether a finding is actionable, needs exception, or can be shipped
+- Staleness warnings for offline snapshots
+
+The Explorer must remain “quiet by default, never silent”: VEX-based suppression should be reversible and auditable, not a destructive delete.
+
+## References
+
+- `docs/VULNERABILITY_EXPLORER_GUIDE.md`
+- `docs/VEX_CONSENSUS_GUIDE.md`
+- `docs/modules/vuln-explorer/architecture.md`

docs/modules/vex-lens/guides/issuer-directory.md

@@ -0,0 +1,33 @@
+# VEX Issuer Directory and Trust
+
+Issuer trust is a first-class input to VEX decisioning. The issuer directory defines *who* a statement is from, *how* it is verified, and *how* policy should weigh it.
+
+## Issuer Identity
+
+An issuer can be identified by:
+
+- Stable provider IDs (connector/provider identifiers)
+- Cryptographic identity (certificate chain, key identifiers, transparency inclusion proof) when available
+
+Tenants may define allowlists for acceptable issuer identities and/or map issuers into trust tiers.
+
+## Trust Tiers and Weights
+
+Trust is commonly expressed as:
+
+- **Tier label** (e.g., vendor, distro, internal, untrusted)
+- **Weight/confidence** used by consensus/policy to break ties and set posture
+- **Verification requirements** (e.g., signature required for `not_affected` gating)
+
+## Offline Trust
+
+Offline deployments must be able to verify issuer identity without network access:
+
+- Trust roots and allowlists are bundled in the Offline Kit
+- Signature verification and transparency proofs are evaluated against bundled material
+
+## References
+
+- `docs/VEX_CONSENSUS_GUIDE.md`
+- `docs/modules/excititor/architecture.md`
+- `docs/modules/vex-lens/architecture.md`

docs/modules/vex-lens/guides/vex-trust-gate-rollout.md

@@ -0,0 +1,223 @@
+# VexTrustGate Rollout Guide
+
+This guide describes the phased rollout procedure for the VexTrustGate policy feature, which enforces VEX signature verification trust thresholds.
+
+## Overview
+
+VexTrustGate adds a new policy gate that:
+1. Validates VEX signature verification trust scores
+2. Enforces per-environment thresholds (production stricter than staging/dev)
+3. Blocks or warns on status transitions when trust is insufficient
+4. Contributes to confidence scoring via VexTrustConfidenceFactorProvider
+
+## Gate Order
+
+VexTrustGate is positioned in the policy gate chain at **order 250**:
+- **100**: EvidenceCompleteness
+- **200**: LatticeState
+- **250**: VexTrust ← NEW
+- **300**: UncertaintyTier
+- **400**: Confidence
+
+## Prerequisites
+
+1. VEX signature verification pipeline active (SPRINT_1227_0004_0001)
+2. IssuerDirectory populated with trusted VEX sources
+3. Excititor properly populating VexTrustStatus in API responses
+
+## Rollout Phases
+
+### Phase 1: Feature Flag Deployment
+
+Deploy with gate disabled to establish baseline:
+
+```yaml
+PolicyGates:
+  VexTrust:
+    Enabled: false  # Gate off initially
+```
+
+**Duration**: 1-2 days
+**Monitoring**: Verify deployment health, no regression in existing gates.
+
+### Phase 2: Shadow Mode (Warn Everywhere)
+
+Enable gate in warn-only mode across all environments:
+
+```yaml
+PolicyGates:
+  VexTrust:
+    Enabled: true
+    Thresholds:
+      production:
+        MinCompositeScore: 0.80
+        RequireIssuerVerified: true
+        FailureAction: Warn  # Changed from Block
+      staging:
+        MinCompositeScore: 0.60
+        RequireIssuerVerified: true
+        FailureAction: Warn
+      development:
+        MinCompositeScore: 0.40
+        RequireIssuerVerified: false
+        FailureAction: Warn
+    MissingTrustBehavior: Warn
+```
+
+**Duration**: 1-2 weeks
+**Monitoring**: 
+- Review `stellaops.policy.vex_trust_gate.decisions.total` metrics
+- Analyze warn events to understand threshold impact
+- Collect feedback from operators on false positives
+
+### Phase 3: Threshold Tuning
+
+Based on Phase 2 data, adjust thresholds:
+
+1. **Review decision breakdown by reason**:
+   - `composite_score`: May need to lower threshold
+   - `issuer_verified`: Check IssuerDirectory completeness
+   - `freshness`: Consider expanding acceptable states
+
+2. **Tenant-specific adjustments** (if needed):
+   ```yaml
+   PolicyGates:
+     VexTrust:
+       TenantOverrides:
+         tenant-with-internal-vex:
+           production:
+             MinCompositeScore: 0.70  # Lower for self-signed internal VEX
+         high-security-tenant:
+           production:
+             MinCompositeScore: 0.90  # Higher for regulated workloads
+   ```
+
+**Duration**: 1 week
+**Outcome**: Validated threshold configuration
+
+### Phase 4: Production Enforcement
+
+Enable blocking in production only:
+
+```yaml
+PolicyGates:
+  VexTrust:
+    Enabled: true
+    Thresholds:
+      production:
+        MinCompositeScore: 0.80
+        RequireIssuerVerified: true
+        MinAccuracyRate: 0.85
+        AcceptableFreshness:
+          - fresh
+        FailureAction: Block  # Now enforcing
+      staging:
+        FailureAction: Warn  # Still warn only
+      development:
+        FailureAction: Warn
+```
+
+**Duration**: Ongoing with monitoring
+**Rollback**: Set `FailureAction: Warn` or `Enabled: false` if issues arise.
+
+### Phase 5: Full Rollout
+
+After production stabilization, optionally enable blocking in staging:
+
+```yaml
+PolicyGates:
+  VexTrust:
+    Thresholds:
+      staging:
+        MinCompositeScore: 0.60
+        RequireIssuerVerified: true
+        FailureAction: Block  # Optional stricter staging
+```
+
+## Monitoring
+
+### Key Metrics
+
+| Metric | Description | Alert Threshold |
+|--------|-------------|-----------------|
+| `stellaops.policy.vex_trust_gate.evaluations.total` | Total evaluations | Baseline variance |
+| `stellaops.policy.vex_trust_gate.decisions.total{decision="block"}` | Block decisions | Sudden spike |
+| `stellaops.policy.vex_trust_gate.trust_score` | Score distribution | Mean < 0.50 |
+| `stellaops.policy.vex_trust_gate.evaluation_duration_ms` | Latency | p99 > 100ms |
+
+### Trace Spans
+
+- `VexTrustGate.EvaluateAsync`
+  - Attributes: `environment`, `trust_score`, `decision`, `issuer_id`
+
+### Audit Trail
+
+PolicyAuditEntity now includes VEX trust fields:
+- `VexTrustScore`: Composite score at decision time
+- `VexTrustTier`: Tier classification
+- `VexSignatureVerified`: Whether signature was verified
+- `VexIssuerId`/`VexIssuerName`: Issuer info
+- `VexTrustGateResult`: Gate decision
+- `VexTrustGateReason`: Reason code
+
+## Rollback Procedure
+
+### Immediate Disable
+```yaml
+PolicyGates:
+  VexTrust:
+    Enabled: false
+```
+
+### Switch to Warn Mode
+```yaml
+PolicyGates:
+  VexTrust:
+    Thresholds:
+      production:
+        FailureAction: Warn
+      staging:
+        FailureAction: Warn
+      development:
+        FailureAction: Warn
+```
+
+### Per-Tenant Disable
+```yaml
+PolicyGates:
+  VexTrust:
+    TenantOverrides:
+      affected-tenant:
+        production:
+          MinCompositeScore: 0.01  # Effectively bypass
+          RequireIssuerVerified: false
+```
+
+## Troubleshooting
+
+### Common Issues
+
+| Symptom | Likely Cause | Resolution |
+|---------|--------------|------------|
+| All VEX blocked | Missing IssuerDirectory entries | Populate directory with trusted issuers |
+| High false positive rate | Threshold too strict | Lower `MinCompositeScore` |
+| "missing_vex_trust_data" warnings | Verification pipeline not running | Check Excititor logs |
+| Inconsistent decisions | Stale trust cache | Verify cache TTL settings |
+
+### Debug Logging
+
+Enable debug logging for gate:
+```yaml
+Logging:
+  LogLevel:
+    StellaOps.Policy.Engine.Gates.VexTrustGate: Debug
+```
+
+## Support
+
+- Sprint: `SPRINT_1227_0004_0003`
+- Component: `StellaOps.Policy.Engine.Gates`
+- Files:
+  - `src/Policy/StellaOps.Policy.Engine/Gates/VexTrustGate.cs`
+  - `src/Policy/StellaOps.Policy.Engine/Gates/VexTrustGateOptions.cs`
+  - `etc/policy-gates.yaml.sample`