Add unit tests for VexLens normalizer, CPE parser, product mapper, and PURL parser
- Implemented comprehensive tests for VexLensNormalizer including format detection and normalization scenarios. - Added tests for CpeParser covering CPE 2.3 and 2.2 formats, invalid inputs, and canonical key generation. - Created tests for ProductMapper to validate parsing and matching logic across different strictness levels. - Developed tests for PurlParser to ensure correct parsing of various PURL formats and validation of identifiers. - Introduced stubs for Monaco editor and worker to facilitate testing in the web application. - Updated project file for the test project to include necessary dependencies.
This commit is contained in:
StellaOps Bot
319
docs/modules/vexlens/architecture.md
Normal file
319
docs/modules/vexlens/architecture.md
Normal file
@@ -0,0 +1,319 @@
|
||||
# component_architecture_vexlens.md — **Stella Ops VexLens** (2025Q4)
|
||||
|
||||
> Supports deliverables from Epic 30 – VEX Consensus Engine and Epic 31 – Advisory AI Integration.
|
||||
|
||||
> **Scope.** Implementation-ready architecture for **VexLens**: the consensus engine for computing authoritative VEX (Vulnerability Exploitability eXchange) status from multiple overlapping statements. It supports trust-weighted voting, lattice-based conflict resolution, and provides policy integration for vulnerability decisioning.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & Boundaries
|
||||
|
||||
**Mission.** Compute deterministic VEX consensus status from multiple sources with full audit trail, enabling automated vulnerability triage based on exploitability data.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* **VexLens does not fetch VEX documents** — it receives normalized statements from Excititor or direct API input.
|
||||
* **VexLens does not store raw VEX documents** — it stores computed projections and consensus results.
|
||||
* **VexLens does not make policy decisions** — it provides VEX status to Policy Engine for final determination.
|
||||
|
||||
---
|
||||
|
||||
## 1) Responsibilities (contract)
|
||||
|
||||
1. **Normalize** VEX documents from OpenVEX, CSAF VEX, CycloneDX VEX, and SPDX VEX formats.
|
||||
2. **Map products** using PURL and CPE identifiers with configurable matching strictness.
|
||||
3. **Verify signatures** on VEX documents (DSSE, JWS, PGP, PKCS#7).
|
||||
4. **Compute trust weights** based on issuer authority, signature status, freshness, and other factors.
|
||||
5. **Compute consensus** using configurable modes:
|
||||
- **HighestWeight**: Single highest-weighted statement wins
|
||||
- **WeightedVote**: Weighted voting among all statements
|
||||
- **Lattice**: Most conservative status wins (affected > under_investigation > not_affected > fixed)
|
||||
- **AuthoritativeFirst**: Authoritative sources override others
|
||||
- **MostRecent**: Most recent statement wins
|
||||
6. **Store projections** for historical tracking and audit.
|
||||
7. **Emit events** on consensus computation, status changes, and conflict detection.
|
||||
8. **Integrate** with Policy Engine for vulnerability suppression and severity adjustment.
|
||||
|
||||
---
|
||||
|
||||
## 2) External Dependencies
|
||||
|
||||
* **Excititor**: Provides normalized VEX statements from connectors.
|
||||
* **Policy Engine**: Consumes VEX consensus for vulnerability decisioning.
|
||||
* **Vuln Explorer**: Enriches vulnerability data with VEX status.
|
||||
* **Orchestrator**: Schedules consensus compute jobs for batch processing.
|
||||
* **Authority**: Validates issuer trust and key fingerprints.
|
||||
* **Config stores**: MongoDB (projections, issuer directory), Redis (caches).
|
||||
|
||||
---
|
||||
|
||||
## 3) API Surface
|
||||
|
||||
Base path: `/api/v1/vexlens`. Full OpenAPI spec at `docs/api/vexlens-openapi.yaml`.
|
||||
|
||||
### 3.1 Consensus Operations
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/consensus` | POST | Compute consensus for a vulnerability-product pair |
|
||||
| `/consensus/batch` | POST | Compute consensus for multiple pairs in batch |
|
||||
|
||||
### 3.2 Projection Queries
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/projections` | GET | Query consensus projections with filtering |
|
||||
| `/projections/{projectionId}` | GET | Get a projection by ID |
|
||||
| `/projections/latest` | GET | Get latest projection for a vuln-product pair |
|
||||
| `/projections/history` | GET | Get projection history |
|
||||
|
||||
### 3.3 Issuer Directory
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/issuers` | GET | List registered issuers |
|
||||
| `/issuers` | POST | Register a new issuer |
|
||||
| `/issuers/{issuerId}` | GET | Get issuer details |
|
||||
| `/issuers/{issuerId}` | DELETE | Revoke an issuer |
|
||||
| `/issuers/{issuerId}/keys` | POST | Add a key to an issuer |
|
||||
| `/issuers/{issuerId}/keys/{fingerprint}` | DELETE | Revoke a key |
|
||||
|
||||
### 3.4 Statistics
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/statistics` | GET | Get consensus statistics |
|
||||
|
||||
---
|
||||
|
||||
## 4) Data Flow
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
|
||||
│ Excititor │────▶│ Normalizer │────▶│ Trust Weighting │
|
||||
│ (VEX Docs) │ │ (OpenVEX, │ │ (9 factors) │
|
||||
└─────────────┘ │ CSAF, CDX) │ └────────┬────────┘
|
||||
└──────────────┘ │
|
||||
▼
|
||||
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
|
||||
│ Policy │◀────│ Projection │◀────│ Consensus │
|
||||
│ Engine │ │ Store │ │ Engine │
|
||||
└─────────────┘ └──────────────┘ └─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Events │
|
||||
│ (Computed, │
|
||||
│ StatusChange,│
|
||||
│ Conflict) │
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5) VEX Status Lattice
|
||||
|
||||
VexLens uses a status lattice for conservative conflict resolution:
|
||||
|
||||
```
|
||||
affected (most restrictive)
|
||||
│
|
||||
▼
|
||||
under_investigation
|
||||
│
|
||||
▼
|
||||
not_affected
|
||||
│
|
||||
▼
|
||||
fixed (least restrictive)
|
||||
```
|
||||
|
||||
In lattice mode, the most restrictive status always wins. This ensures that when sources disagree, the system errs on the side of caution.
|
||||
|
||||
---
|
||||
|
||||
## 6) Trust Weight Factors
|
||||
|
||||
| Factor | Weight | Description |
|
||||
|--------|--------|-------------|
|
||||
| IssuerBase | 25% | Base trust from issuer directory |
|
||||
| SignatureStatus | 15% | Valid/invalid/unsigned signature |
|
||||
| Freshness | 15% | Document age with exponential decay |
|
||||
| IssuerCategory | 10% | Vendor > Distributor > Aggregator |
|
||||
| IssuerTier | 10% | Authoritative > Trusted > Untrusted |
|
||||
| StatusQuality | 10% | Has justification, specific status |
|
||||
| TransparencyLog | 5% | Sigstore Rekor entry |
|
||||
| SourceMatch | 5% | Source URI pattern match |
|
||||
| ProductAuthority | 5% | Issuer is authoritative for product |
|
||||
|
||||
---
|
||||
|
||||
## 7) Configuration
|
||||
|
||||
```yaml
|
||||
vexlens:
|
||||
consensus:
|
||||
defaultMode: WeightedVote # HighestWeight, WeightedVote, Lattice, AuthoritativeFirst, MostRecent
|
||||
minimumConfidence: 0.1
|
||||
conflictThreshold: 0.3
|
||||
requireJustificationForNotAffected: false
|
||||
trust:
|
||||
freshnessHalfLifeDays: 90
|
||||
minimumFreshness: 0.3
|
||||
allowUnsigned: true
|
||||
unsignedPenalty: 0.3
|
||||
allowUnknownIssuers: true
|
||||
unknownIssuerPenalty: 0.5
|
||||
storage:
|
||||
projectionRetentionDays: 365
|
||||
eventRetentionDays: 90
|
||||
issuerDirectory:
|
||||
source: mongodb # mongodb, file, api
|
||||
refreshIntervalMinutes: 60
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8) Storage Schema
|
||||
|
||||
### 8.1 Consensus Projection
|
||||
|
||||
```json
|
||||
{
|
||||
"projectionId": "proj-abc123",
|
||||
"vulnerabilityId": "CVE-2024-1234",
|
||||
"productKey": "pkg:npm/lodash@4.17.21",
|
||||
"tenantId": "tenant-001",
|
||||
"status": "not_affected",
|
||||
"justification": "vulnerable_code_not_present",
|
||||
"confidenceScore": 0.95,
|
||||
"outcome": "Unanimous",
|
||||
"statementCount": 3,
|
||||
"conflictCount": 0,
|
||||
"rationaleSummary": "Unanimous consensus from 3 authoritative sources",
|
||||
"computedAt": "2025-12-06T12:00:00Z",
|
||||
"storedAt": "2025-12-06T12:00:01Z",
|
||||
"previousProjectionId": null,
|
||||
"statusChanged": true
|
||||
}
|
||||
```
|
||||
|
||||
### 8.2 Issuer Record
|
||||
|
||||
```json
|
||||
{
|
||||
"issuerId": "npm-security",
|
||||
"name": "npm Security Team",
|
||||
"category": "Vendor",
|
||||
"trustTier": "Authoritative",
|
||||
"status": "Active",
|
||||
"keyFingerprints": [
|
||||
{
|
||||
"fingerprint": "ABCD1234EFGH5678",
|
||||
"keyType": "Pgp",
|
||||
"algorithm": "EdDSA",
|
||||
"status": "Active",
|
||||
"registeredAt": "2025-01-01T00:00:00Z",
|
||||
"expiresAt": null
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"description": "Official npm security advisories",
|
||||
"uri": "https://www.npmjs.com/advisories",
|
||||
"email": "security@npmjs.com"
|
||||
},
|
||||
"registeredAt": "2025-01-01T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9) Events
|
||||
|
||||
### 9.1 ConsensusComputedEvent
|
||||
|
||||
Emitted after every consensus computation.
|
||||
|
||||
```json
|
||||
{
|
||||
"eventId": "evt-abc123",
|
||||
"projectionId": "proj-abc123",
|
||||
"vulnerabilityId": "CVE-2024-1234",
|
||||
"productKey": "pkg:npm/lodash@4.17.21",
|
||||
"status": "not_affected",
|
||||
"confidenceScore": 0.95,
|
||||
"outcome": "Unanimous",
|
||||
"statementCount": 3,
|
||||
"computedAt": "2025-12-06T12:00:00Z",
|
||||
"emittedAt": "2025-12-06T12:00:01Z"
|
||||
}
|
||||
```
|
||||
|
||||
### 9.2 ConsensusStatusChangedEvent
|
||||
|
||||
Emitted when consensus status changes from previous projection.
|
||||
|
||||
### 9.3 ConsensusConflictDetectedEvent
|
||||
|
||||
Emitted when conflicts are detected during consensus computation.
|
||||
|
||||
---
|
||||
|
||||
## 10) Observability
|
||||
|
||||
### 10.1 Metrics (OpenTelemetry)
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `vexlens.consensus.computed_total` | Counter | Total consensus computations |
|
||||
| `vexlens.consensus.conflicts_total` | Counter | Total conflicts detected |
|
||||
| `vexlens.consensus.confidence` | Histogram | Confidence score distribution |
|
||||
| `vexlens.consensus.duration_seconds` | Histogram | Computation duration |
|
||||
| `vexlens.consensus.status_changes_total` | Counter | Status changes detected |
|
||||
| `vexlens.normalization.documents_total` | Counter | Documents normalized |
|
||||
| `vexlens.trust.weight_value` | Histogram | Trust weight distribution |
|
||||
| `vexlens.issuer.registered_total` | Counter | Issuers registered |
|
||||
|
||||
### 10.2 Traces
|
||||
|
||||
Activity source: `StellaOps.VexLens`
|
||||
|
||||
| Activity | Description |
|
||||
|----------|-------------|
|
||||
| `vexlens.normalize` | VEX document normalization |
|
||||
| `vexlens.compute_trust_weight` | Trust weight computation |
|
||||
| `vexlens.compute_consensus` | Consensus computation |
|
||||
| `vexlens.store_projection` | Projection storage |
|
||||
| `vexlens.query_projections` | Projection query |
|
||||
|
||||
### 10.3 Logging
|
||||
|
||||
Structured logging with event IDs in `VexLensLogEvents`:
|
||||
- 1xxx: Normalization events
|
||||
- 2xxx: Product mapping events
|
||||
- 3xxx: Signature verification events
|
||||
- 4xxx: Trust weight events
|
||||
- 5xxx: Consensus events
|
||||
- 6xxx: Projection events
|
||||
- 7xxx: Issuer directory events
|
||||
|
||||
---
|
||||
|
||||
## 11) Security Considerations
|
||||
|
||||
1. **Issuer Trust**: All issuers must be registered with verified key fingerprints.
|
||||
2. **Signature Verification**: Documents should be cryptographically signed for production use.
|
||||
3. **Tenant Isolation**: Projections are scoped to tenants; no cross-tenant data access.
|
||||
4. **Audit Trail**: All consensus computations are logged with full rationale.
|
||||
5. **Determinism**: All computations are deterministic for reproducibility.
|
||||
|
||||
---
|
||||
|
||||
## 12) Test Matrix
|
||||
|
||||
| Test Category | Coverage | Notes |
|
||||
|---------------|----------|-------|
|
||||
| Unit tests | Normalizer, Parser, Trust, Consensus | 89+ tests |
|
||||
| Determinism harness | Normalization, Trust, Consensus | Verify reproducibility |
|
||||
| Integration tests | API service, Storage, Events | End-to-end flows |
|
||||
| Property-based tests | Lattice semantics, Weight computation | Invariant verification |
|
||||
475
docs/modules/vexlens/operations/deployment.md
Normal file
475
docs/modules/vexlens/operations/deployment.md
Normal file
@@ -0,0 +1,475 @@
|
||||
# VexLens Deployment Runbook
|
||||
|
||||
> Operational runbook for deploying and configuring VexLens consensus engine.
|
||||
|
||||
---
|
||||
|
||||
## 1) Prerequisites
|
||||
|
||||
### 1.1 Infrastructure Requirements
|
||||
|
||||
| Component | Requirement | Notes |
|
||||
|-----------|-------------|-------|
|
||||
| Runtime | .NET 10.0+ | LTS recommended |
|
||||
| Database | MongoDB 6.0+ | For projections and issuer directory |
|
||||
| Cache | Redis 7.0+ (optional) | For caching consensus results |
|
||||
| Memory | 512MB minimum | 2GB recommended for production |
|
||||
| CPU | 2 cores minimum | 4 cores for high throughput |
|
||||
|
||||
### 1.2 Dependencies
|
||||
|
||||
- **Excititor**: VEX document ingestion service
|
||||
- **Authority**: OIDC token validation
|
||||
- **Policy Engine**: (optional) For VEX-aware policy evaluation
|
||||
|
||||
---
|
||||
|
||||
## 2) Configuration
|
||||
|
||||
### 2.1 Environment Variables
|
||||
|
||||
```bash
|
||||
# Core Settings
|
||||
VEXLENS_CONSENSUS_DEFAULT_MODE=WeightedVote
|
||||
VEXLENS_CONSENSUS_MINIMUM_CONFIDENCE=0.1
|
||||
VEXLENS_CONSENSUS_CONFLICT_THRESHOLD=0.3
|
||||
|
||||
# Trust Settings
|
||||
VEXLENS_TRUST_FRESHNESS_HALFLIFE_DAYS=90
|
||||
VEXLENS_TRUST_MINIMUM_FRESHNESS=0.3
|
||||
VEXLENS_TRUST_ALLOW_UNSIGNED=true
|
||||
VEXLENS_TRUST_UNSIGNED_PENALTY=0.3
|
||||
VEXLENS_TRUST_ALLOW_UNKNOWN_ISSUERS=true
|
||||
VEXLENS_TRUST_UNKNOWN_ISSUER_PENALTY=0.5
|
||||
|
||||
# Storage
|
||||
VEXLENS_STORAGE_MONGODB_CONNECTION_STRING=mongodb://localhost:27017
|
||||
VEXLENS_STORAGE_MONGODB_DATABASE=vexlens
|
||||
VEXLENS_STORAGE_PROJECTION_RETENTION_DAYS=365
|
||||
VEXLENS_STORAGE_EVENT_RETENTION_DAYS=90
|
||||
|
||||
# Issuer Directory
|
||||
VEXLENS_ISSUER_DIRECTORY_SOURCE=mongodb
|
||||
VEXLENS_ISSUER_DIRECTORY_REFRESH_INTERVAL_MINUTES=60
|
||||
|
||||
# Observability
|
||||
VEXLENS_OTEL_EXPORTER_ENDPOINT=http://otel-collector:4317
|
||||
VEXLENS_OTEL_SERVICE_NAME=vexlens
|
||||
```
|
||||
|
||||
### 2.2 Configuration File (vexlens.yaml)
|
||||
|
||||
```yaml
|
||||
vexlens:
|
||||
consensus:
|
||||
defaultMode: WeightedVote
|
||||
minimumConfidence: 0.1
|
||||
conflictThreshold: 0.3
|
||||
requireJustificationForNotAffected: false
|
||||
|
||||
trust:
|
||||
freshnessHalfLifeDays: 90
|
||||
minimumFreshness: 0.3
|
||||
allowUnsigned: true
|
||||
unsignedPenalty: 0.3
|
||||
allowUnknownIssuers: true
|
||||
unknownIssuerPenalty: 0.5
|
||||
factorWeights:
|
||||
IssuerBase: 0.25
|
||||
SignatureStatus: 0.15
|
||||
Freshness: 0.15
|
||||
IssuerCategory: 0.10
|
||||
IssuerTier: 0.10
|
||||
StatusQuality: 0.10
|
||||
TransparencyLog: 0.05
|
||||
SourceMatch: 0.05
|
||||
ProductAuthority: 0.05
|
||||
|
||||
storage:
|
||||
mongodb:
|
||||
connectionString: mongodb://localhost:27017
|
||||
database: vexlens
|
||||
projectionsCollection: consensus_projections
|
||||
issuersCollection: issuers
|
||||
projectionRetentionDays: 365
|
||||
eventRetentionDays: 90
|
||||
|
||||
issuerDirectory:
|
||||
source: mongodb
|
||||
refreshIntervalMinutes: 60
|
||||
seedFile: /etc/vexlens/issuers.json
|
||||
|
||||
observability:
|
||||
metrics:
|
||||
enabled: true
|
||||
exporterEndpoint: http://otel-collector:4317
|
||||
tracing:
|
||||
enabled: true
|
||||
samplingRatio: 0.1
|
||||
logging:
|
||||
level: Information
|
||||
format: json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3) Deployment Steps
|
||||
|
||||
### 3.1 Docker Deployment
|
||||
|
||||
```bash
|
||||
# Pull the image
|
||||
docker pull stellaops/vexlens:latest
|
||||
|
||||
# Run with configuration
|
||||
docker run -d \
|
||||
--name vexlens \
|
||||
-p 8080:8080 \
|
||||
-v /etc/vexlens:/etc/vexlens:ro \
|
||||
-e VEXLENS_STORAGE_MONGODB_CONNECTION_STRING=mongodb://mongo:27017 \
|
||||
stellaops/vexlens:latest
|
||||
```
|
||||
|
||||
### 3.2 Kubernetes Deployment
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: vexlens
|
||||
namespace: stellaops
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: vexlens
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: vexlens
|
||||
spec:
|
||||
containers:
|
||||
- name: vexlens
|
||||
image: stellaops/vexlens:latest
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
env:
|
||||
- name: VEXLENS_STORAGE_MONGODB_CONNECTION_STRING
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: vexlens-secrets
|
||||
key: mongodb-connection-string
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "500m"
|
||||
limits:
|
||||
memory: "2Gi"
|
||||
cpu: "2000m"
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health/live
|
||||
port: 8080
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 30
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health/ready
|
||||
port: 8080
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/vexlens
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: vexlens-config
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: vexlens
|
||||
namespace: stellaops
|
||||
spec:
|
||||
selector:
|
||||
app: vexlens
|
||||
ports:
|
||||
- port: 80
|
||||
targetPort: 8080
|
||||
```
|
||||
|
||||
### 3.3 Helm Deployment
|
||||
|
||||
```bash
|
||||
helm install vexlens stellaops/vexlens \
|
||||
--namespace stellaops \
|
||||
--set mongodb.connectionString=mongodb://mongo:27017 \
|
||||
--set replicas=2 \
|
||||
--set resources.requests.memory=512Mi \
|
||||
--set resources.limits.memory=2Gi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4) Issuer Directory Setup
|
||||
|
||||
### 4.1 Seed Issuers File
|
||||
|
||||
Create `/etc/vexlens/issuers.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"issuers": [
|
||||
{
|
||||
"issuerId": "npm-security",
|
||||
"name": "npm Security Team",
|
||||
"category": "Vendor",
|
||||
"trustTier": "Authoritative",
|
||||
"keyFingerprints": [
|
||||
{
|
||||
"fingerprint": "ABCD1234EFGH5678",
|
||||
"keyType": "Pgp",
|
||||
"algorithm": "EdDSA"
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"description": "Official npm security advisories",
|
||||
"uri": "https://www.npmjs.com/advisories"
|
||||
}
|
||||
},
|
||||
{
|
||||
"issuerId": "github-security",
|
||||
"name": "GitHub Security Lab",
|
||||
"category": "Aggregator",
|
||||
"trustTier": "Trusted",
|
||||
"metadata": {
|
||||
"description": "GitHub Security Advisories",
|
||||
"uri": "https://github.com/advisories"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 Register Issuer via API
|
||||
|
||||
```bash
|
||||
curl -X POST http://vexlens:8080/api/v1/vexlens/issuers \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-StellaOps-Tenant: tenant-001" \
|
||||
-d '{
|
||||
"issuerId": "vendor-acme",
|
||||
"name": "ACME Corporation",
|
||||
"category": "Vendor",
|
||||
"trustTier": "Authoritative",
|
||||
"initialKeys": [
|
||||
{
|
||||
"fingerprint": "1234ABCD5678EFGH",
|
||||
"keyType": "Pgp",
|
||||
"algorithm": "RSA"
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"description": "ACME security advisories",
|
||||
"uri": "https://security.acme.example.com"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5) Health Checks
|
||||
|
||||
### 5.1 Liveness Probe
|
||||
|
||||
```bash
|
||||
curl http://vexlens:8080/health/live
|
||||
# Response: {"status": "Healthy"}
|
||||
```
|
||||
|
||||
### 5.2 Readiness Probe
|
||||
|
||||
```bash
|
||||
curl http://vexlens:8080/health/ready
|
||||
# Response: {"status": "Healthy", "checks": {"mongodb": "Healthy", "issuerDirectory": "Healthy"}}
|
||||
```
|
||||
|
||||
### 5.3 Detailed Health
|
||||
|
||||
```bash
|
||||
curl http://vexlens:8080/health/detailed
|
||||
# Full health check with component details
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6) Monitoring
|
||||
|
||||
### 6.1 Key Metrics to Monitor
|
||||
|
||||
| Metric | Alert Threshold | Description |
|
||||
|--------|-----------------|-------------|
|
||||
| `vexlens.consensus.duration_seconds` | p99 > 5s | Consensus computation latency |
|
||||
| `vexlens.consensus.conflicts_total` | rate > 100/min | High conflict rate |
|
||||
| `vexlens.normalization.errors_total` | rate > 10/min | Normalization failures |
|
||||
| `vexlens.projection.query_duration_seconds` | p99 > 1s | Slow projection queries |
|
||||
|
||||
### 6.2 Grafana Dashboard
|
||||
|
||||
Import the VexLens dashboard from `deploy/grafana/vexlens-dashboard.json`.
|
||||
|
||||
### 6.3 Alerting Rules
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
- name: vexlens
|
||||
rules:
|
||||
- alert: VexLensHighLatency
|
||||
expr: histogram_quantile(0.99, rate(vexlens_consensus_duration_seconds_bucket[5m])) > 5
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "VexLens consensus latency is high"
|
||||
|
||||
- alert: VexLensHighConflictRate
|
||||
expr: rate(vexlens_consensus_conflicts_total[5m]) > 100
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "VexLens detecting high conflict rate"
|
||||
|
||||
- alert: VexLensNormalizationErrors
|
||||
expr: rate(vexlens_normalization_errors_total[5m]) > 10
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "VexLens normalization errors increasing"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7) Backup and Recovery
|
||||
|
||||
### 7.1 Backup Projections
|
||||
|
||||
```bash
|
||||
# MongoDB backup
|
||||
mongodump --uri="mongodb://localhost:27017" \
|
||||
--db=vexlens \
|
||||
--collection=consensus_projections \
|
||||
--out=/backup/vexlens-$(date +%Y%m%d)
|
||||
```
|
||||
|
||||
### 7.2 Backup Issuer Directory
|
||||
|
||||
```bash
|
||||
# Export issuers to JSON
|
||||
curl http://vexlens:8080/api/v1/vexlens/issuers?limit=1000 \
|
||||
> /backup/issuers-$(date +%Y%m%d).json
|
||||
```
|
||||
|
||||
### 7.3 Restore
|
||||
|
||||
```bash
|
||||
# Restore MongoDB
|
||||
mongorestore --uri="mongodb://localhost:27017" \
|
||||
--db=vexlens \
|
||||
/backup/vexlens-20251206/
|
||||
|
||||
# Re-seed issuers if needed
|
||||
# Issuers are automatically loaded from seed file on startup
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8) Scaling
|
||||
|
||||
### 8.1 Horizontal Scaling
|
||||
|
||||
VexLens is stateless for compute operations. Scale horizontally by adding replicas:
|
||||
|
||||
```bash
|
||||
kubectl scale deployment vexlens --replicas=4 -n stellaops
|
||||
```
|
||||
|
||||
### 8.2 Performance Tuning
|
||||
|
||||
```yaml
|
||||
# For high-throughput deployments
|
||||
vexlens:
|
||||
consensus:
|
||||
# Enable batch processing
|
||||
batchSize: 100
|
||||
batchTimeoutMs: 50
|
||||
|
||||
storage:
|
||||
mongodb:
|
||||
# Connection pool
|
||||
maxConnectionPoolSize: 100
|
||||
minConnectionPoolSize: 10
|
||||
|
||||
caching:
|
||||
enabled: true
|
||||
redis:
|
||||
connectionString: redis://redis:6379
|
||||
consensusTtlMinutes: 5
|
||||
issuerTtlMinutes: 60
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9) Troubleshooting
|
||||
|
||||
### 9.1 Common Issues
|
||||
|
||||
| Issue | Cause | Resolution |
|
||||
|-------|-------|------------|
|
||||
| Slow consensus | Many statements | Enable caching, increase batch size |
|
||||
| High conflict rate | Inconsistent sources | Review issuer trust tiers |
|
||||
| Normalization failures | Invalid VEX format | Check Excititor connector config |
|
||||
| Low confidence scores | Missing signatures | Configure issuer keys |
|
||||
|
||||
### 9.2 Debug Logging
|
||||
|
||||
```bash
|
||||
# Enable debug logging
|
||||
export VEXLENS_OBSERVABILITY_LOGGING_LEVEL=Debug
|
||||
```
|
||||
|
||||
### 9.3 Determinism Verification
|
||||
|
||||
```bash
|
||||
# Run determinism harness
|
||||
curl -X POST http://vexlens:8080/api/v1/vexlens/test/determinism \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"vexContent": "..."}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10) Upgrade Procedure
|
||||
|
||||
### 10.1 Rolling Upgrade
|
||||
|
||||
```bash
|
||||
# Update image
|
||||
kubectl set image deployment/vexlens vexlens=stellaops/vexlens:v1.2.0 -n stellaops
|
||||
|
||||
# Monitor rollout
|
||||
kubectl rollout status deployment/vexlens -n stellaops
|
||||
```
|
||||
|
||||
### 10.2 Database Migrations
|
||||
|
||||
VexLens uses automatic schema migrations. No manual intervention required for minor versions.
|
||||
|
||||
For major version upgrades:
|
||||
1. Backup all data
|
||||
2. Review migration notes in release changelog
|
||||
3. Apply migrations: `vexlens migrate --apply`
|
||||
4. Verify: `vexlens migrate --verify`
|
||||
408
docs/modules/vexlens/operations/offline-kit.md
Normal file
408
docs/modules/vexlens/operations/offline-kit.md
Normal file
@@ -0,0 +1,408 @@
|
||||
# VexLens Offline Kit
|
||||
|
||||
> Air-gapped deployment guide for VexLens consensus engine.
|
||||
|
||||
---
|
||||
|
||||
## 1) Overview
|
||||
|
||||
VexLens can operate in fully air-gapped environments with pre-loaded VEX data and issuer directories. This guide covers offline deployment, bundle creation, and operational procedures.
|
||||
|
||||
---
|
||||
|
||||
## 2) Offline Bundle Structure
|
||||
|
||||
### 2.1 Bundle Manifest
|
||||
|
||||
```json
|
||||
{
|
||||
"bundleId": "vexlens-bundle-2025-12-06",
|
||||
"version": "1.0.0",
|
||||
"createdAt": "2025-12-06T00:00:00Z",
|
||||
"createdBy": "stellaops-export",
|
||||
"checksum": "sha256:abc123...",
|
||||
"components": {
|
||||
"issuerDirectory": {
|
||||
"file": "issuers.json",
|
||||
"checksum": "sha256:def456...",
|
||||
"count": 150
|
||||
},
|
||||
"vexStatements": {
|
||||
"file": "vex-statements.ndjson.gz",
|
||||
"checksum": "sha256:ghi789...",
|
||||
"count": 50000
|
||||
},
|
||||
"projectionSnapshots": {
|
||||
"file": "projections.ndjson.gz",
|
||||
"checksum": "sha256:jkl012...",
|
||||
"count": 25000
|
||||
},
|
||||
"trustConfiguration": {
|
||||
"file": "trust-config.yaml",
|
||||
"checksum": "sha256:mno345..."
|
||||
}
|
||||
},
|
||||
"compatibility": {
|
||||
"minVersion": "1.0.0",
|
||||
"maxVersion": "2.0.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 Bundle Contents
|
||||
|
||||
```
|
||||
vexlens-bundle-2025-12-06/
|
||||
├── manifest.json
|
||||
├── issuers.json
|
||||
├── vex-statements.ndjson.gz
|
||||
├── projections.ndjson.gz
|
||||
├── trust-config.yaml
|
||||
├── checksums.sha256
|
||||
└── signature.dsse
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3) Creating Offline Bundles
|
||||
|
||||
### 3.1 Export Command
|
||||
|
||||
```bash
|
||||
# Export from online VexLens instance
|
||||
stellaops vexlens export \
|
||||
--output /export/vexlens-bundle-$(date +%Y-%m-%d) \
|
||||
--include-issuers \
|
||||
--include-statements \
|
||||
--include-projections \
|
||||
--compress \
|
||||
--sign
|
||||
```
|
||||
|
||||
### 3.2 Selective Export
|
||||
|
||||
```bash
|
||||
# Export only specific tenants
|
||||
stellaops vexlens export \
|
||||
--output /export/tenant-bundle \
|
||||
--tenant tenant-001,tenant-002 \
|
||||
--since 2025-01-01 \
|
||||
--compress
|
||||
|
||||
# Export only critical vulnerabilities
|
||||
stellaops vexlens export \
|
||||
--output /export/critical-bundle \
|
||||
--vulnerability-pattern "CVE-202[45]-*" \
|
||||
--status affected,under_investigation \
|
||||
--compress
|
||||
```
|
||||
|
||||
### 3.3 Bundle Signing
|
||||
|
||||
```bash
|
||||
# Sign bundle with organization key
|
||||
stellaops vexlens export sign \
|
||||
--bundle /export/vexlens-bundle-2025-12-06 \
|
||||
--key /keys/export-signing-key.pem \
|
||||
--output /export/vexlens-bundle-2025-12-06/signature.dsse
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4) Importing Offline Bundles
|
||||
|
||||
### 4.1 Verification
|
||||
|
||||
```bash
|
||||
# Verify bundle integrity and signature
|
||||
stellaops vexlens import verify \
|
||||
--bundle /import/vexlens-bundle-2025-12-06 \
|
||||
--trust-root /etc/vexlens/trust-roots.pem
|
||||
|
||||
# Output:
|
||||
# Bundle ID: vexlens-bundle-2025-12-06
|
||||
# Created: 2025-12-06T00:00:00Z
|
||||
# Signature: VALID (signed by: StellaOps Export Service)
|
||||
# Checksums: VALID (all 4 files verified)
|
||||
# Compatibility: COMPATIBLE (current version: 1.1.0)
|
||||
```
|
||||
|
||||
### 4.2 Import Command
|
||||
|
||||
```bash
|
||||
# Import bundle to offline VexLens
|
||||
stellaops vexlens import \
|
||||
--bundle /import/vexlens-bundle-2025-12-06 \
|
||||
--mode merge \
|
||||
--verify-signature
|
||||
|
||||
# Import modes:
|
||||
# - merge: Add new data, keep existing
|
||||
# - replace: Replace all data with bundle contents
|
||||
# - incremental: Only add data newer than existing
|
||||
```
|
||||
|
||||
### 4.3 Staged Import
|
||||
|
||||
For large bundles, use staged import:
|
||||
|
||||
```bash
|
||||
# Stage 1: Import issuers
|
||||
stellaops vexlens import \
|
||||
--bundle /import/bundle \
|
||||
--component issuer-directory \
|
||||
--dry-run
|
||||
|
||||
# Stage 2: Import statements
|
||||
stellaops vexlens import \
|
||||
--bundle /import/bundle \
|
||||
--component vex-statements \
|
||||
--batch-size 1000
|
||||
|
||||
# Stage 3: Import projections
|
||||
stellaops vexlens import \
|
||||
--bundle /import/bundle \
|
||||
--component projections \
|
||||
--batch-size 5000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5) Offline Configuration
|
||||
|
||||
### 5.1 Air-Gap Mode Settings
|
||||
|
||||
```yaml
|
||||
vexlens:
|
||||
airgap:
|
||||
enabled: true
|
||||
# Disable external connectivity checks
|
||||
allowExternalConnections: false
|
||||
# Use file-based issuer directory
|
||||
issuerDirectorySource: file
|
||||
# Pre-compute consensus on import
|
||||
precomputeConsensus: true
|
||||
|
||||
trust:
|
||||
# Stricter settings for air-gap
|
||||
allowUnsigned: false
|
||||
allowUnknownIssuers: false
|
||||
# Use local trust anchors
|
||||
trustAnchors: /etc/vexlens/trust-anchors.pem
|
||||
|
||||
storage:
|
||||
# Local storage only
|
||||
mongodb:
|
||||
connectionString: mongodb://localhost:27017
|
||||
# No external cache
|
||||
redis:
|
||||
enabled: false
|
||||
|
||||
time:
|
||||
# Use time anchor for staleness checks
|
||||
timeAnchorFile: /etc/vexlens/time-anchor.json
|
||||
# Maximum allowed drift
|
||||
maxDriftDays: 7
|
||||
```
|
||||
|
||||
### 5.2 Time Anchor Configuration
|
||||
|
||||
For air-gapped environments, use time anchors:
|
||||
|
||||
```json
|
||||
{
|
||||
"anchorTime": "2025-12-06T00:00:00Z",
|
||||
"signature": "base64...",
|
||||
"validUntil": "2025-12-13T00:00:00Z",
|
||||
"signedBy": "stellaops-time-authority"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6) Operational Procedures
|
||||
|
||||
### 6.1 Bundle Update Cycle
|
||||
|
||||
1. **Export** (Online environment):
|
||||
```bash
|
||||
stellaops vexlens export --output /export/weekly-bundle --compress --sign
|
||||
```
|
||||
|
||||
2. **Transfer** (Secure media):
|
||||
- Copy bundle to removable media
|
||||
- Verify checksums after transfer
|
||||
- Log transfer in custody chain
|
||||
|
||||
3. **Verify** (Offline environment):
|
||||
```bash
|
||||
stellaops vexlens import verify --bundle /import/weekly-bundle
|
||||
```
|
||||
|
||||
4. **Import** (Offline environment):
|
||||
```bash
|
||||
stellaops vexlens import --bundle /import/weekly-bundle --mode incremental
|
||||
```
|
||||
|
||||
5. **Recompute** (If needed):
|
||||
```bash
|
||||
stellaops vexlens consensus recompute --since $(date -d '7 days ago' +%Y-%m-%d)
|
||||
```
|
||||
|
||||
### 6.2 Staleness Monitoring
|
||||
|
||||
```bash
|
||||
# Check data freshness
|
||||
stellaops vexlens status --staleness
|
||||
|
||||
# Output:
|
||||
# Data Freshness Report
|
||||
# ---------------------
|
||||
# Issuer Directory: 2 days old (OK)
|
||||
# VEX Statements: 5 days old (OK)
|
||||
# Projections: 5 days old (OK)
|
||||
# Time Anchor: 2 days old (OK)
|
||||
#
|
||||
# Overall Status: FRESH
|
||||
```
|
||||
|
||||
### 6.3 Audit Trail
|
||||
|
||||
All import operations are logged:
|
||||
|
||||
```bash
|
||||
# View import history
|
||||
stellaops vexlens import history --limit 10
|
||||
|
||||
# Output:
|
||||
# Import History
|
||||
# --------------
|
||||
# 2025-12-06 08:00: vexlens-bundle-2025-12-06 (merge, 50000 statements)
|
||||
# 2025-11-29 08:00: vexlens-bundle-2025-11-29 (incremental, 12000 statements)
|
||||
# ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7) Degraded Mode Operation
|
||||
|
||||
### 7.1 Degradation Matrix
|
||||
|
||||
| Component | Degradation | Impact | Mitigation |
|
||||
|-----------|-------------|--------|------------|
|
||||
| Stale VEX data | >7 days old | Lower accuracy | Schedule bundle update |
|
||||
| Missing issuers | Unknown issuer | Lower trust scores | Add issuer to directory |
|
||||
| No projections | Cold start | Slower first queries | Pre-compute on import |
|
||||
| Time drift | >24 hours | Staleness warnings | Update time anchor |
|
||||
|
||||
### 7.2 Emergency Recovery
|
||||
|
||||
If bundle import fails:
|
||||
|
||||
```bash
|
||||
# Check bundle integrity
|
||||
stellaops vexlens import verify --bundle /import/bundle --verbose
|
||||
|
||||
# Attempt partial import
|
||||
stellaops vexlens import --bundle /import/bundle --skip-corrupted
|
||||
|
||||
# Rollback to previous state
|
||||
stellaops vexlens import rollback --to vexlens-bundle-2025-11-29
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8) Bundle Management
|
||||
|
||||
### 8.1 Retention Policy
|
||||
|
||||
```yaml
|
||||
vexlens:
|
||||
bundles:
|
||||
# Keep last N bundles
|
||||
retentionCount: 5
|
||||
# Minimum age before deletion
|
||||
minimumAgeDays: 30
|
||||
# Archive location
|
||||
archivePath: /archive/vexlens-bundles
|
||||
```
|
||||
|
||||
### 8.2 Storage Requirements
|
||||
|
||||
| Data Type | Typical Size | Compression Ratio |
|
||||
|-----------|--------------|-------------------|
|
||||
| Issuers | 1-5 MB | 5:1 |
|
||||
| Statements | 100-500 MB | 10:1 |
|
||||
| Projections | 50-200 MB | 8:1 |
|
||||
| **Total Bundle** | **150-700 MB** | **8:1** |
|
||||
|
||||
### 8.3 Bundle Cleanup
|
||||
|
||||
```bash
|
||||
# Clean old bundles
|
||||
stellaops vexlens bundles cleanup --keep 5
|
||||
|
||||
# Archive bundles older than 30 days
|
||||
stellaops vexlens bundles archive --older-than 30d --to /archive
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9) Security Considerations
|
||||
|
||||
### 9.1 Bundle Signing
|
||||
|
||||
All bundles should be signed before transfer:
|
||||
|
||||
```bash
|
||||
# Verify signature chain
|
||||
stellaops vexlens import verify-chain \
|
||||
--bundle /import/bundle \
|
||||
--trust-root /etc/vexlens/root-ca.pem
|
||||
```
|
||||
|
||||
### 9.2 Transfer Security
|
||||
|
||||
1. Use encrypted removable media
|
||||
2. Maintain custody chain documentation
|
||||
3. Verify checksums at each transfer point
|
||||
4. Log all bundle operations
|
||||
|
||||
### 9.3 Access Control
|
||||
|
||||
```yaml
|
||||
vexlens:
|
||||
security:
|
||||
# Require authentication for import
|
||||
importRequiresAuth: true
|
||||
# Allowed import roles
|
||||
importRoles: [vexlens.admin, vexlens.operator]
|
||||
# Audit all imports
|
||||
auditImports: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10) Troubleshooting
|
||||
|
||||
### 10.1 Common Issues
|
||||
|
||||
| Issue | Cause | Resolution |
|
||||
|-------|-------|------------|
|
||||
| Import fails | Corrupted bundle | Re-export from source |
|
||||
| Signature invalid | Wrong trust root | Update trust anchors |
|
||||
| Time anchor expired | Stale time anchor | Generate new anchor |
|
||||
| Missing issuers | Incomplete export | Include issuers in export |
|
||||
|
||||
### 10.2 Diagnostic Commands
|
||||
|
||||
```bash
|
||||
# Verify bundle contents
|
||||
stellaops vexlens bundle inspect /import/bundle
|
||||
|
||||
# Check import readiness
|
||||
stellaops vexlens import preflight --bundle /import/bundle
|
||||
|
||||
# Generate diagnostic report
|
||||
stellaops vexlens diagnostics --output /tmp/diag.json
|
||||
```
|
||||
Reference in New Issue
Block a user