git.stella-ops.org/docs/guides/epss-integration-v4.md

# EPSS v4 Integration Guide

## Overview

EPSS (Exploit Prediction Scoring System) v4 is a machine learning-based vulnerability scoring system developed by FIRST.org that predicts the probability a CVE will be exploited in the wild within the next 30 days. StellaOps integrates EPSS as a **probabilistic threat signal** alongside CVSS v4's **deterministic severity assessment**, enabling more accurate vulnerability prioritization.

**Key Concepts**:
- **EPSS Score**: Probability (0.0-1.0) that a CVE will be exploited in next 30 days
- **EPSS Percentile**: Ranking (0.0-1.0) of this CVE relative to all scored CVEs
- **Model Date**: Date for which EPSS scores were computed
- **Immutable at-scan**: EPSS evidence captured at scan time never changes (deterministic replay)
- **Current EPSS**: Live projection for triage (updated daily)

---

## How EPSS Works

EPSS uses machine learning to predict exploitation probability based on:

1. **Vulnerability Characteristics**: CVSS metrics, CWE, affected products
2. **Social Signals**: Twitter/GitHub mentions, security blog posts
3. **Exploit Database Entries**: Exploit-DB, Metasploit, etc.
4. **Historical Exploitation**: Past exploitation patterns

EPSS is updated **daily** by FIRST.org based on fresh threat intelligence.

### EPSS vs CVSS

| Dimension | CVSS v4 | EPSS v4 |
|-----------|---------|---------|
| **Nature** | Deterministic severity | Probabilistic threat |
| **Scale** | 0.0-10.0 (severity) | 0.0-1.0 (probability) |
| **Update Frequency** | Static (per CVE version) | Daily (live threat data) |
| **Purpose** | Impact assessment | Likelihood assessment |
| **Source** | Vendor/NVD | FIRST.org ML model |

**Example**:
- **CVE-2024-1234**: CVSS 9.8 (Critical) + EPSS 0.01 (1st percentile)
  - Interpretation: Severe impact if exploited, but very unlikely to be exploited
  - Priority: **Medium** (deprioritize despite high CVSS)

- **CVE-2024-5678**: CVSS 6.5 (Medium) + EPSS 0.95 (98th percentile)
  - Interpretation: Moderate impact, but actively being exploited
  - Priority: **High** (escalate despite moderate CVSS)

---

## Architecture Overview

### Data Flow

```
┌────────────────────────────────────────────────────────────────┐
│                   EPSS Data Lifecycle in StellaOps              │
└────────────────────────────────────────────────────────────────┘

1. INGESTION (Daily 00:05 UTC)
   ┌───────────────────┐
   │   FIRST.org       │ Daily CSV: epss_scores-YYYY-MM-DD.csv.gz
   │  (300k CVEs)      │ ~15MB compressed
   └────────┬──────────┘
            │
            ▼
   ┌───────────────────────────────────────────────────────────┐
   │ Concelier: EpssIngestJob                                   │
   │ - Download/Import CSV                                      │
   │ - Parse (handle # comment, validate bounds)                │
   │ - Bulk insert: epss_scores (partitioned by month)          │
   │ - Compute delta: epss_changes (flags for enrichment)       │
   │ - Upsert: epss_current (latest projection)                 │
   │ - Emit event: "epss.updated"                               │
   └────────┬──────────────────────────────────────────────────┘
            │
            ▼
   [PostgreSQL: concelier.epss_*]
            │
            ├─────────────────────────────┐
            │                             │
            ▼                             ▼

2. AT-SCAN CAPTURE (Immutable Evidence)
   ┌────────────────────────────────────────────────────────────┐
   │ Scanner: On new scan                                        │
   │ - Bulk query: epss_current for CVE list                    │
   │ - Store immutable evidence:                                 │
   │   * epss_score_at_scan                                      │
   │   * epss_percentile_at_scan                                 │
   │   * epss_model_date_at_scan                                 │
   │   * epss_import_run_id_at_scan                              │
   │ - Use in lattice decision (SR→CR if EPSS≥90th)              │
   └─────────────────────────────────────────────────────────────┘

3. LIVE ENRICHMENT (Existing Findings)
   ┌─────────────────────────────────────────────────────────────┐
   │ Concelier: EpssEnrichmentJob (on "epss.updated")            │
   │ - Read: epss_changes WHERE flags IN (CROSSED_HIGH, BIG_JUMP)│
   │ - Find impacted: vuln_instance_triage BY cve_id             │
   │ - Update: current_epss_score, current_epss_percentile       │
   │ - If priority band changed → emit "vuln.priority.changed"   │
   └────────┬────────────────────────────────────────────────────┘
            │
            ▼
   ┌─────────────────────────────────────────────────────────────┐
   │ Notify: On "vuln.priority.changed"                          │
   │ - Check tenant notification rules                           │
   │ - Send: Slack / Email / Teams / In-app                      │
   │ - Payload: EPSS delta, threshold crossed                    │
   └─────────────────────────────────────────────────────────────┘

4. POLICY SCORING
   ┌─────────────────────────────────────────────────────────────┐
   │ Policy Engine: Risk Score Formula                           │
   │ risk_score = (cvss/10) + epss_bonus + kev_bonus + reach_mult│
   │                                                              │
   │ EPSS Bonus (Simple Profile):                                │
   │   - Percentile ≥99th: +10%                                  │
   │   - Percentile ≥90th: +5%                                   │
   │   - Percentile ≥50th: +2%                                   │
   │   - Percentile <50th: 0%                                    │
   │                                                              │
   │ VEX Lattice Rules:                                          │
   │   - SR + EPSS≥90th → Escalate to CR (Confirmed Reachable)  │
   │   - DV + EPSS≥95th → Flag for review (vendor denial)       │
   │   - U + EPSS≥95th → Prioritize for reachability analysis   │
   └─────────────────────────────────────────────────────────────┘
```

### Database Schema

**Location**: `concelier` database

#### epss_import_runs (Provenance)

Tracks each EPSS import with full provenance for audit trail.

```sql
CREATE TABLE concelier.epss_import_runs (
  import_run_id        UUID PRIMARY KEY,
  model_date           DATE NOT NULL UNIQUE,
  source_uri           TEXT NOT NULL,
  file_sha256          TEXT NOT NULL,
  row_count            INT NOT NULL,
  model_version_tag    TEXT NULL,
  published_date       DATE NULL,
  status               TEXT NOT NULL,  -- IN_PROGRESS, SUCCEEDED, FAILED
  created_at           TIMESTAMPTZ NOT NULL
);
```

#### epss_scores (Time-Series, Partitioned)

Immutable append-only history of daily EPSS scores.

```sql
CREATE TABLE concelier.epss_scores (
  model_date       DATE NOT NULL,
  cve_id           TEXT NOT NULL,
  epss_score       DOUBLE PRECISION NOT NULL,
  percentile       DOUBLE PRECISION NOT NULL,
  import_run_id    UUID NOT NULL,
  PRIMARY KEY (model_date, cve_id)
) PARTITION BY RANGE (model_date);
```

**Partitions**: Monthly (e.g., `epss_scores_2025_12`)

#### epss_current (Latest Projection)

Materialized view of latest EPSS score per CVE for fast lookups.

```sql
CREATE TABLE concelier.epss_current (
  cve_id            TEXT PRIMARY KEY,
  epss_score        DOUBLE PRECISION NOT NULL,
  percentile        DOUBLE PRECISION NOT NULL,
  model_date        DATE NOT NULL,
  import_run_id     UUID NOT NULL,
  updated_at        TIMESTAMPTZ NOT NULL
);
```

**Usage**: Scanner bulk queries this table for new scans.

#### epss_changes (Delta Tracking, Partitioned)

Tracks material EPSS changes for targeted enrichment.

```sql
CREATE TABLE concelier.epss_changes (
  model_date       DATE NOT NULL,
  cve_id           TEXT NOT NULL,
  old_score        DOUBLE PRECISION NULL,
  new_score        DOUBLE PRECISION NOT NULL,
  delta_score      DOUBLE PRECISION NULL,
  old_percentile   DOUBLE PRECISION NULL,
  new_percentile   DOUBLE PRECISION NOT NULL,
  delta_percentile DOUBLE PRECISION NULL,
  flags            INT NOT NULL,  -- Bitmask
  PRIMARY KEY (model_date, cve_id)
) PARTITION BY RANGE (model_date);
```

**Flags** (bitmask):
- `1` = NEW_SCORED (CVE newly appeared)
- `2` = CROSSED_HIGH (percentile ≥95th)
- `4` = BIG_JUMP (|Δscore| ≥0.10)
- `8` = DROPPED_LOW (percentile <50th)
- `16` = SCORE_INCREASED
- `32` = SCORE_DECREASED

---

## Configuration

### Scheduler Configuration

**File**: `etc/scheduler.yaml`

```yaml
scheduler:
  jobs:
    - name: epss.ingest
      schedule: "0 5 0 * * *"  # Daily at 00:05 UTC
      worker: concelier
      args:
        source: online
        date: null  # Auto: yesterday
      timeout: 600s
      retry:
        max_attempts: 3
        backoff: exponential
```

### Concelier Configuration

**File**: `etc/concelier.yaml`

```yaml
concelier:
  epss:
    enabled: true
    online_source:
      base_url: "https://epss.empiricalsecurity.com/"
      url_pattern: "epss_scores-{date:yyyy-MM-dd}.csv.gz"
      timeout: 180s
    bundle_source:
      path: "/opt/stellaops/bundles/epss/"
    thresholds:
      high_percentile: 0.95     # Top 5%
      high_score: 0.50          # 50% probability
      big_jump_delta: 0.10      # 10 percentage points
      low_percentile: 0.50      # Median
    enrichment:
      enabled: true
      batch_size: 1000
      flags_to_process:
        - NEW_SCORED
        - CROSSED_HIGH
        - BIG_JUMP
```

### Scanner Configuration

**File**: `etc/scanner.yaml`

```yaml
scanner:
  epss:
    enabled: true
    provider: postgres
    cache_ttl: 3600
    fallback_on_missing: unknown  # Options: unknown, zero, skip
```

### Policy Configuration

**File**: `etc/policy.yaml`

```yaml
policy:
  scoring:
    epss:
      enabled: true
      profile: simple  # Options: simple, advanced, custom
      simple_bonuses:
        percentile_99: 0.10  # +10%
        percentile_90: 0.05  # +5%
        percentile_50: 0.02  # +2%
  lattice:
    epss_escalation:
      enabled: true
      sr_to_cr_threshold: 0.90  # SR→CR if EPSS≥90th percentile
```

---

## Daily Operation

### Automated Ingestion

EPSS data is ingested automatically daily at **00:05 UTC** via Scheduler.

**Workflow**:
1. Scheduler triggers `epss.ingest` job at 00:05 UTC
2. Concelier downloads `epss_scores-YYYY-MM-DD.csv.gz` from FIRST.org
3. CSV parsed (comment line → metadata, rows → scores)
4. Bulk insert into `epss_scores` partition (NpgsqlBinaryImporter)
5. Compute delta: `epss_changes` (compare vs `epss_current`)
6. Upsert `epss_current` (latest projection)
7. Emit `epss.updated` event
8. Enrichment job updates impacted vulnerability instances
9. Notifications sent if priority bands changed

**Monitoring**:
```bash
# Check latest model date
stellaops epss status

# Output:
# EPSS Status:
#   Latest Model Date: 2025-12-16
#   Import Time: 2025-12-17 00:07:32 UTC
#   CVE Count: 231,417
#   Staleness: FRESH (1 day)
```

### Manual Triggering

```bash
# Trigger manual ingest (force re-import)
stellaops concelier job trigger epss.ingest --date 2025-12-16 --force

# Backfill historical data (last 30 days)
stellaops epss backfill --from 2025-11-17 --to 2025-12-16
```

---

## Air-Gapped Operation

### Bundle Structure

EPSS data for offline deployments is packaged in risk bundles:

```
risk-bundle-2025-12-16/
├── manifest.json
├── epss/
│   ├── epss_scores-2025-12-16.csv.zst      # ZSTD compressed
│   └── epss_metadata.json
├── kev/
│   └── kev-catalog.json
└── signatures/
    └── bundle.dsse.json
```

### EPSS Metadata

**File**: `epss/epss_metadata.json`

```json
{
  "model_date": "2025-12-16",
  "model_version": "v2025.12.16",
  "published_date": "2025-12-16",
  "row_count": 231417,
  "sha256": "abc123...",
  "source_uri": "https://epss.empiricalsecurity.com/epss_scores-2025-12-16.csv.gz",
  "created_at": "2025-12-16T00:00:00Z"
}
```

### Import Procedure

```bash
# 1. Transfer bundle to air-gapped system
scp risk-bundle-2025-12-16.tar.zst airgap-host:/opt/stellaops/bundles/

# 2. Import bundle
stellaops offline import --bundle /opt/stellaops/bundles/risk-bundle-2025-12-16.tar.zst

# 3. Verify import
stellaops epss status

# Output:
# EPSS Status:
#   Latest Model Date: 2025-12-16
#   Source: bundle://risk-bundle-2025-12-16
#   CVE Count: 231,417
#   Staleness: ACCEPTABLE (within 7 days)
```

### Update Cadence

**Recommended**:
- **Online**: Daily (automatic)
- **Air-gapped**: Weekly (manual bundle import)

**Staleness Thresholds**:
- **FRESH**: ≤1 day
- **ACCEPTABLE**: ≤7 days
- **STALE**: ≤14 days
- **VERY_STALE**: >14 days (alert, fallback to CVSS-only)

---

## Scanner Integration

### EPSS Evidence in Scan Findings

Every scan finding includes **immutable EPSS-at-scan** evidence:

```json
{
  "finding_id": "CVE-2024-12345-pkg:npm/lodash@4.17.21",
  "cve_id": "CVE-2024-12345",
  "product": "pkg:npm/lodash@4.17.21",
  "scan_id": "scan-abc123",
  "scan_timestamp": "2025-12-17T10:30:00Z",
  "evidence": {
    "cvss_v4": {
      "vector_string": "CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:H/VI:H/VA:H/SC:H/SI:H/SA:H",
      "base_score": 9.3,
      "severity": "CRITICAL"
    },
    "epss_at_scan": {
      "epss_score": 0.42357,
      "percentile": 0.88234,
      "model_date": "2025-12-16",
      "import_run_id": "550e8400-e29b-41d4-a716-446655440000"
    },
    "epss_current": {
      "epss_score": 0.45123,
      "percentile": 0.89456,
      "model_date": "2025-12-17",
      "delta_score": 0.02766,
      "delta_percentile": 0.01222,
      "trend": "RISING"
    }
  }
}
```

**Key Points**:
- **epss_at_scan**: Immutable, captured at scan time (deterministic replay)
- **epss_current**: Mutable, updated daily for live triage
- **Replay**: Historical scans always use `epss_at_scan` for consistent policy evaluation

### Bulk Query Optimization

Scanner queries EPSS for all CVEs in a single database call:

```sql
SELECT cve_id, epss_score, percentile, model_date, import_run_id
FROM concelier.epss_current
WHERE cve_id = ANY(@cve_ids);
```

**Performance**: <500ms for 10k CVEs (P95)

---

## Policy Engine Integration

### Risk Score Formula

**Simple Profile**:

```
risk_score = (cvss_base / 10) + epss_bonus + kev_bonus
```

**EPSS Bonus Table**:

| EPSS Percentile | Bonus | Rationale |
|----------------|-------|-----------|
| ≥99th | +10% | Top 1% most likely to be exploited |
| ≥90th | +5% | Top 10% high exploitation probability |
| ≥50th | +2% | Above median moderate risk |
| <50th | 0% | Below median no bonus |

**Advanced Profile**:

Adds:
- **KEV synergy**: If in KEV catalog → multiply EPSS bonus by 1.5
- **Uncertainty penalty**: Missing EPSS → -5%
- **Temporal decay**: EPSS >30 days stale → reduce bonus by 50%

### VEX Lattice Rules

**Escalation**:
- **SR (Static Reachable) + EPSS≥90th** → Auto-escalate to **CR (Confirmed Reachable)**
- Rationale: High exploit probability warrants confirmation

**Review Flags**:
- **DV (Denied by Vendor VEX) + EPSS≥95th** → Flag for manual review
- Rationale: Vendor denial contradicted by active exploitation signals

**Prioritization**:
- **U (Unknown) + EPSS≥95th** → Prioritize for reachability analysis
- Rationale: High exploit probability justifies effort

### SPL (Stella Policy Language) Syntax

```yaml
# Custom policy using EPSS
rules:
  - name: high_epss_escalation
    condition: |
      epss.percentile >= 0.95 AND
      lattice.state == "SR" AND
      runtime.exposed == true
    action: escalate_to_cr
    reason: "High EPSS (top 5%) + Static Reachable + Runtime Exposed"

  - name: epss_trend_alert
    condition: |
      epss.delta_score >= 0.10 AND
      cvss.base_score >= 7.0
    action: notify
    channels: [slack, email]
    reason: "EPSS jumped by 10+ points (was {epss.old_score}, now {epss.new_score})"
```

**Available Fields**:
- `epss.score` - Current EPSS score (0.0-1.0)
- `epss.percentile` - Current percentile (0.0-1.0)
- `epss.model_date` - Model date
- `epss.delta_score` - Change vs previous scan
- `epss.trend` - RISING, FALLING, STABLE
- `epss.at_scan.score` - Immutable score at scan time
- `epss.at_scan.percentile` - Immutable percentile at scan time

---

## Notification Integration

### Event: vuln.priority.changed

Emitted when EPSS change causes priority band shift.

**Payload**:

```json
{
  "event_type": "vuln.priority.changed",
  "vulnerability_id": "CVE-2024-12345",
  "product_key": "pkg:npm/lodash@4.17.21",
  "old_priority_band": "medium",
  "new_priority_band": "high",
  "reason": "EPSS percentile crossed 95th (was 88th, now 96th)",
  "epss_change": {
    "old_score": 0.42,
    "new_score": 0.78,
    "delta_score": 0.36,
    "old_percentile": 0.88,
    "new_percentile": 0.96,
    "model_date": "2025-12-16"
  }
}
```

### Notification Rules

**File**: `etc/notify.yaml`

```yaml
notify:
  rules:
    - name: epss_crossed_high
      event_type: vuln.priority.changed
      condition: "payload.epss_change.new_percentile >= 0.95"
      channels: [slack, email]
      template: epss_high_alert
      digest: false  # Immediate

    - name: epss_big_jump
      event_type: vuln.priority.changed
      condition: "payload.epss_change.delta_score >= 0.10"
      channels: [slack]
      template: epss_rising_threat
      digest: true
      digest_time: "09:00"  # Daily digest at 9 AM
```

### Slack Template Example

```
🚨 **High EPSS Alert**

**CVE**: CVE-2024-12345
**Product**: pkg:npm/lodash@4.17.21
**EPSS**: 0.78 (96th percentile) ⬆️ from 0.42 (88th percentile)
**Delta**: +0.36 (36 percentage points)
**Priority**: Medium → **High**

**Action Required**: Review and prioritize remediation.

[View in StellaOps →](https://stellaops.example.com/vulns/CVE-2024-12345)
```

---

## Troubleshooting

### EPSS Data Not Available

**Symptom**: Scans show "EPSS: N/A"

**Diagnosis**:
```bash
# Check EPSS status
stellaops epss status

# Check import runs
stellaops concelier jobs list --type epss.ingest --limit 10
```

**Resolution**:
1. **No imports**: Trigger manual ingest
   ```bash
   stellaops concelier job trigger epss.ingest
   ```

2. **Import failed**: Check logs
   ```bash
   stellaops concelier logs --job-id <id> --level ERROR
   ```

3. **FIRST.org down**: Use air-gapped bundle
   ```bash
   stellaops offline import --bundle /path/to/risk-bundle.tar.zst
   ```

### Stale EPSS Data

**Symptom**: UI shows "EPSS stale (14 days)"

**Diagnosis**:
```sql
SELECT * FROM concelier.epss_model_staleness;
-- Output: days_stale: 14, staleness_status: STALE
```

**Resolution**:
1. **Online**: Check scheduler job status
   ```bash
   stellaops scheduler jobs status epss.ingest
   ```

2. **Air-gapped**: Import fresh bundle
   ```bash
   stellaops offline import --bundle /path/to/latest-bundle.tar.zst
   ```

3. **Fallback**: Disable EPSS temporarily (uses CVSS-only)
   ```yaml
   # etc/scanner.yaml
   scanner:
     epss:
       enabled: false
   ```

### High Memory Usage During Ingest

**Symptom**: Concelier worker OOM during EPSS ingest

**Diagnosis**:
```bash
# Check memory metrics
stellaops metrics query 'process_resident_memory_bytes{service="concelier"}'
```

**Resolution**:
1. **Increase worker memory limit**:
   ```yaml
   # Kubernetes deployment
   resources:
     limits:
       memory: 1Gi  # Was 512Mi
   ```

2. **Verify streaming parser** (should not load full CSV into memory):
   ```bash
   # Check logs for "EPSS CSV parsed: rows_yielded="
   stellaops concelier logs --job-type epss.ingest | grep "CSV parsed"
   ```

---

## Best Practices

### 1. Combine Signals (Never Use EPSS Alone)

❌ **Don't**: `if epss > 0.95 then CRITICAL`

✅ **Do**: `if cvss >= 8.0 AND epss >= 0.95 AND runtime_exposed then CRITICAL`

### 2. Review High EPSS Manually

Manually review vulnerabilities with EPSS ≥95th percentile, especially if:
- CVSS is low (<7.0) but EPSS is high
- Vendor VEX denies exploitability but EPSS is high

### 3. Track Trends

Monitor EPSS changes over time:
- Rising EPSS → increasing threat
- Falling EPSS → threat subsiding

### 4. Update Regularly

- **Online**: Daily (automatic)
- **Air-gapped**: Weekly minimum, daily preferred

### 5. Verify During Audits

For compliance audits, use EPSS-at-scan (immutable) not current EPSS:
```sql
SELECT epss_score_at_scan, epss_model_date_at_scan
FROM scan_findings
WHERE scan_id = 'audit-scan-20251217';
```

---

## API Reference

### Query Current EPSS

```bash
# Single CVE
stellaops epss get CVE-2024-12345

# Output:
# CVE-2024-12345
#   Score: 0.42357 (42.4% probability)
#   Percentile: 88.2th
#   Model Date: 2025-12-16
#   Status: FRESH
```

### Batch Query

```bash
# From file
stellaops epss batch --file cves.txt --output epss-scores.json

# cves.txt:
# CVE-2024-1
# CVE-2024-2
# CVE-2024-3
```

### Query History

```bash
# Last 180 days
stellaops epss history CVE-2024-12345 --days 180 --format csv

# Output: epss-history-CVE-2024-12345.csv
# model_date,epss_score,percentile
# 2025-12-17,0.45123,0.89456
# 2025-12-16,0.42357,0.88234
# ...
```

### Top CVEs by EPSS

```bash
# Top 100
stellaops epss top --limit 100 --format table

# Output:
# Rank | CVE           | Score  | Percentile | CVSS
# -----|---------------|--------|------------|------
# 1    | CVE-2024-9999 | 0.9872 | 99.9th     | 9.8
# 2    | CVE-2024-8888 | 0.9654 | 99.8th     | 8.1
# ...
```

---

## References

- **FIRST EPSS Homepage**: https://www.first.org/epss/
- **EPSS Data & Stats**: https://www.first.org/epss/data_stats
- **EPSS API Docs**: https://www.first.org/epss/api
- **CVSS v4.0 Spec**: https://www.first.org/cvss/v4.0/specification-document
- **StellaOps Policy Guide**: `docs/policy/overview.md`
- **StellaOps Reachability Guide**: `docs/modules/scanner/reachability.md`

---

**Last Updated**: 2025-12-17
**Version**: 1.0
**Maintainer**: StellaOps Security Team