Add unit tests for VexLens normalizer, CPE parser, product mapper, and PURL parser

- Implemented comprehensive tests for VexLensNormalizer including format detection and normalization scenarios. - Added tests for CpeParser covering CPE 2.3 and 2.2 formats, invalid inputs, and canonical key generation. - Created tests for ProductMapper to validate parsing and matching logic across different strictness levels. - Developed tests for PurlParser to ensure correct parsing of various PURL formats and validation of identifiers. - Introduced stubs for Monaco editor and worker to facilitate testing in the web application. - Updated project file for the test project to include necessary dependencies.
2025-12-06 16:28:12 +02:00
parent 2b892ad1b2
commit efd6850c38
132 changed files with 16675 additions and 5428 deletions
--- a/docs/modules/vexlens/architecture.md
+++ b/docs/modules/vexlens/architecture.md
@@ -0,0 +1,319 @@
+# component_architecture_vexlens.md — **Stella Ops VexLens** (2025Q4)
+
+> Supports deliverables from Epic 30 – VEX Consensus Engine and Epic 31 – Advisory AI Integration.
+
+> **Scope.** Implementation-ready architecture for **VexLens**: the consensus engine for computing authoritative VEX (Vulnerability Exploitability eXchange) status from multiple overlapping statements. It supports trust-weighted voting, lattice-based conflict resolution, and provides policy integration for vulnerability decisioning.
+
+---
+
+## 0) Mission & Boundaries
+
+**Mission.** Compute deterministic VEX consensus status from multiple sources with full audit trail, enabling automated vulnerability triage based on exploitability data.
+
+**Boundaries.**
+
+* **VexLens does not fetch VEX documents** — it receives normalized statements from Excititor or direct API input.
+* **VexLens does not store raw VEX documents** — it stores computed projections and consensus results.
+* **VexLens does not make policy decisions** — it provides VEX status to Policy Engine for final determination.
+
+---
+
+## 1) Responsibilities (contract)
+
+1. **Normalize** VEX documents from OpenVEX, CSAF VEX, CycloneDX VEX, and SPDX VEX formats.
+2. **Map products** using PURL and CPE identifiers with configurable matching strictness.
+3. **Verify signatures** on VEX documents (DSSE, JWS, PGP, PKCS#7).
+4. **Compute trust weights** based on issuer authority, signature status, freshness, and other factors.
+5. **Compute consensus** using configurable modes:
+   - **HighestWeight**: Single highest-weighted statement wins
+   - **WeightedVote**: Weighted voting among all statements
+   - **Lattice**: Most conservative status wins (affected > under_investigation > not_affected > fixed)
+   - **AuthoritativeFirst**: Authoritative sources override others
+   - **MostRecent**: Most recent statement wins
+6. **Store projections** for historical tracking and audit.
+7. **Emit events** on consensus computation, status changes, and conflict detection.
+8. **Integrate** with Policy Engine for vulnerability suppression and severity adjustment.
+
+---
+
+## 2) External Dependencies
+
+* **Excititor**: Provides normalized VEX statements from connectors.
+* **Policy Engine**: Consumes VEX consensus for vulnerability decisioning.
+* **Vuln Explorer**: Enriches vulnerability data with VEX status.
+* **Orchestrator**: Schedules consensus compute jobs for batch processing.
+* **Authority**: Validates issuer trust and key fingerprints.
+* **Config stores**: MongoDB (projections, issuer directory), Redis (caches).
+
+---
+
+## 3) API Surface
+
+Base path: `/api/v1/vexlens`. Full OpenAPI spec at `docs/api/vexlens-openapi.yaml`.
+
+### 3.1 Consensus Operations
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/consensus` | POST | Compute consensus for a vulnerability-product pair |
+| `/consensus/batch` | POST | Compute consensus for multiple pairs in batch |
+
+### 3.2 Projection Queries
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/projections` | GET | Query consensus projections with filtering |
+| `/projections/{projectionId}` | GET | Get a projection by ID |
+| `/projections/latest` | GET | Get latest projection for a vuln-product pair |
+| `/projections/history` | GET | Get projection history |
+
+### 3.3 Issuer Directory
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/issuers` | GET | List registered issuers |
+| `/issuers` | POST | Register a new issuer |
+| `/issuers/{issuerId}` | GET | Get issuer details |
+| `/issuers/{issuerId}` | DELETE | Revoke an issuer |
+| `/issuers/{issuerId}/keys` | POST | Add a key to an issuer |
+| `/issuers/{issuerId}/keys/{fingerprint}` | DELETE | Revoke a key |
+
+### 3.4 Statistics
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/statistics` | GET | Get consensus statistics |
+
+---
+
+## 4) Data Flow
+
+```
+┌─────────────┐     ┌──────────────┐     ┌─────────────────┐
+│  Excititor  │────▶│  Normalizer  │────▶│ Trust Weighting │
+│ (VEX Docs)  │     │  (OpenVEX,   │     │  (9 factors)    │
+└─────────────┘     │ CSAF, CDX)   │     └────────┬────────┘
+                    └──────────────┘              │
+                                                  ▼
+┌─────────────┐     ┌──────────────┐     ┌─────────────────┐
+│   Policy    │◀────│ Projection   │◀────│   Consensus     │
+│   Engine    │     │   Store      │     │    Engine       │
+└─────────────┘     └──────────────┘     └─────────────────┘
+                           │
+                           ▼
+                    ┌──────────────┐
+                    │    Events    │
+                    │ (Computed,   │
+                    │ StatusChange,│
+                    │ Conflict)    │
+                    └──────────────┘
+```
+
+---
+
+## 5) VEX Status Lattice
+
+VexLens uses a status lattice for conservative conflict resolution:
+
+```
+affected (most restrictive)
+    │
+    ▼
+under_investigation
+    │
+    ▼
+not_affected
+    │
+    ▼
+fixed (least restrictive)
+```
+
+In lattice mode, the most restrictive status always wins. This ensures that when sources disagree, the system errs on the side of caution.
+
+---
+
+## 6) Trust Weight Factors
+
+| Factor | Weight | Description |
+|--------|--------|-------------|
+| IssuerBase | 25% | Base trust from issuer directory |
+| SignatureStatus | 15% | Valid/invalid/unsigned signature |
+| Freshness | 15% | Document age with exponential decay |
+| IssuerCategory | 10% | Vendor > Distributor > Aggregator |
+| IssuerTier | 10% | Authoritative > Trusted > Untrusted |
+| StatusQuality | 10% | Has justification, specific status |
+| TransparencyLog | 5% | Sigstore Rekor entry |
+| SourceMatch | 5% | Source URI pattern match |
+| ProductAuthority | 5% | Issuer is authoritative for product |
+
+---
+
+## 7) Configuration
+
+```yaml
+vexlens:
+  consensus:
+    defaultMode: WeightedVote  # HighestWeight, WeightedVote, Lattice, AuthoritativeFirst, MostRecent
+    minimumConfidence: 0.1
+    conflictThreshold: 0.3
+    requireJustificationForNotAffected: false
+  trust:
+    freshnessHalfLifeDays: 90
+    minimumFreshness: 0.3
+    allowUnsigned: true
+    unsignedPenalty: 0.3
+    allowUnknownIssuers: true
+    unknownIssuerPenalty: 0.5
+  storage:
+    projectionRetentionDays: 365
+    eventRetentionDays: 90
+  issuerDirectory:
+    source: mongodb  # mongodb, file, api
+    refreshIntervalMinutes: 60
+```
+
+---
+
+## 8) Storage Schema
+
+### 8.1 Consensus Projection
+
+```json
+{
+  "projectionId": "proj-abc123",
+  "vulnerabilityId": "CVE-2024-1234",
+  "productKey": "pkg:npm/lodash@4.17.21",
+  "tenantId": "tenant-001",
+  "status": "not_affected",
+  "justification": "vulnerable_code_not_present",
+  "confidenceScore": 0.95,
+  "outcome": "Unanimous",
+  "statementCount": 3,
+  "conflictCount": 0,
+  "rationaleSummary": "Unanimous consensus from 3 authoritative sources",
+  "computedAt": "2025-12-06T12:00:00Z",
+  "storedAt": "2025-12-06T12:00:01Z",
+  "previousProjectionId": null,
+  "statusChanged": true
+}
+```
+
+### 8.2 Issuer Record
+
+```json
+{
+  "issuerId": "npm-security",
+  "name": "npm Security Team",
+  "category": "Vendor",
+  "trustTier": "Authoritative",
+  "status": "Active",
+  "keyFingerprints": [
+    {
+      "fingerprint": "ABCD1234EFGH5678",
+      "keyType": "Pgp",
+      "algorithm": "EdDSA",
+      "status": "Active",
+      "registeredAt": "2025-01-01T00:00:00Z",
+      "expiresAt": null
+    }
+  ],
+  "metadata": {
+    "description": "Official npm security advisories",
+    "uri": "https://www.npmjs.com/advisories",
+    "email": "security@npmjs.com"
+  },
+  "registeredAt": "2025-01-01T00:00:00Z"
+}
+```
+
+---
+
+## 9) Events
+
+### 9.1 ConsensusComputedEvent
+
+Emitted after every consensus computation.
+
+```json
+{
+  "eventId": "evt-abc123",
+  "projectionId": "proj-abc123",
+  "vulnerabilityId": "CVE-2024-1234",
+  "productKey": "pkg:npm/lodash@4.17.21",
+  "status": "not_affected",
+  "confidenceScore": 0.95,
+  "outcome": "Unanimous",
+  "statementCount": 3,
+  "computedAt": "2025-12-06T12:00:00Z",
+  "emittedAt": "2025-12-06T12:00:01Z"
+}
+```
+
+### 9.2 ConsensusStatusChangedEvent
+
+Emitted when consensus status changes from previous projection.
+
+### 9.3 ConsensusConflictDetectedEvent
+
+Emitted when conflicts are detected during consensus computation.
+
+---
+
+## 10) Observability
+
+### 10.1 Metrics (OpenTelemetry)
+
+| Metric | Type | Description |
+|--------|------|-------------|
+| `vexlens.consensus.computed_total` | Counter | Total consensus computations |
+| `vexlens.consensus.conflicts_total` | Counter | Total conflicts detected |
+| `vexlens.consensus.confidence` | Histogram | Confidence score distribution |
+| `vexlens.consensus.duration_seconds` | Histogram | Computation duration |
+| `vexlens.consensus.status_changes_total` | Counter | Status changes detected |
+| `vexlens.normalization.documents_total` | Counter | Documents normalized |
+| `vexlens.trust.weight_value` | Histogram | Trust weight distribution |
+| `vexlens.issuer.registered_total` | Counter | Issuers registered |
+
+### 10.2 Traces
+
+Activity source: `StellaOps.VexLens`
+
+| Activity | Description |
+|----------|-------------|
+| `vexlens.normalize` | VEX document normalization |
+| `vexlens.compute_trust_weight` | Trust weight computation |
+| `vexlens.compute_consensus` | Consensus computation |
+| `vexlens.store_projection` | Projection storage |
+| `vexlens.query_projections` | Projection query |
+
+### 10.3 Logging
+
+Structured logging with event IDs in `VexLensLogEvents`:
+- 1xxx: Normalization events
+- 2xxx: Product mapping events
+- 3xxx: Signature verification events
+- 4xxx: Trust weight events
+- 5xxx: Consensus events
+- 6xxx: Projection events
+- 7xxx: Issuer directory events
+
+---
+
+## 11) Security Considerations
+
+1. **Issuer Trust**: All issuers must be registered with verified key fingerprints.
+2. **Signature Verification**: Documents should be cryptographically signed for production use.
+3. **Tenant Isolation**: Projections are scoped to tenants; no cross-tenant data access.
+4. **Audit Trail**: All consensus computations are logged with full rationale.
+5. **Determinism**: All computations are deterministic for reproducibility.
+
+---
+
+## 12) Test Matrix
+
+| Test Category | Coverage | Notes |
+|---------------|----------|-------|
+| Unit tests | Normalizer, Parser, Trust, Consensus | 89+ tests |
+| Determinism harness | Normalization, Trust, Consensus | Verify reproducibility |
+| Integration tests | API service, Storage, Events | End-to-end flows |
+| Property-based tests | Lattice semantics, Weight computation | Invariant verification |