git.stella-ops.org/docs/airgap/epss-bundles.md

# EPSS Air-Gapped Bundles Guide

## Overview

This guide describes how to create, distribute, and import EPSS (Exploit Prediction Scoring System) data bundles for air-gapped StellaOps deployments. EPSS bundles enable offline vulnerability risk scoring with the same probabilistic threat intelligence available to online deployments.

**Key Concepts**:
- **Risk Bundle**: Aggregated security data (EPSS + KEV + advisories) for offline import
- **EPSS Snapshot**: Single-day EPSS scores for all CVEs (~300k rows)
- **Staleness Threshold**: How old EPSS data can be before fallback to CVSS-only
- **Deterministic Import**: Same bundle imported twice yields identical database state

---

## Bundle Structure

### Standard Risk Bundle Layout

```
risk-bundle-2025-12-17/
├── manifest.json                           # Bundle metadata and checksums
├── epss/
│   ├── epss_scores-2025-12-17.csv.zst      # EPSS data (ZSTD compressed)
│   └── epss_metadata.json                  # EPSS provenance
├── kev/
│   └── kev-catalog.json                    # CISA KEV catalog
├── advisories/
│   ├── nvd-updates.ndjson.zst
│   └── ghsa-updates.ndjson.zst
└── signatures/
    ├── bundle.dsse.json                    # DSSE signature (optional)
    └── bundle.sha256sums                   # File integrity checksums
```

### manifest.json

```json
{
  "bundle_id": "risk-bundle-2025-12-17",
  "created_at": "2025-12-17T00:00:00Z",
  "created_by": "stellaops-bundler-v1.2.3",
  "bundle_type": "risk",
  "schema_version": "v1",
  "contents": {
    "epss": {
      "model_date": "2025-12-17",
      "file": "epss/epss_scores-2025-12-17.csv.zst",
      "sha256": "abc123...",
      "size_bytes": 15728640,
      "row_count": 231417
    },
    "kev": {
      "catalog_version": "2025-12-17",
      "file": "kev/kev-catalog.json",
      "sha256": "def456...",
      "known_exploited_count": 1247
    },
    "advisories": {
      "nvd": {
        "file": "advisories/nvd-updates.ndjson.zst",
        "sha256": "ghi789...",
        "record_count": 1523
      },
      "ghsa": {
        "file": "advisories/ghsa-updates.ndjson.zst",
        "sha256": "jkl012...",
        "record_count": 8734
      }
    }
  },
  "signature": {
    "type": "dsse",
    "file": "signatures/bundle.dsse.json",
    "key_id": "stellaops-bundler-2025",
    "algorithm": "ed25519"
  }
}
```

### epss/epss_metadata.json

```json
{
  "model_date": "2025-12-17",
  "model_version": "v2025.12.17",
  "published_date": "2025-12-17",
  "row_count": 231417,
  "source_uri": "https://epss.empiricalsecurity.com/epss_scores-2025-12-17.csv.gz",
  "retrieved_at": "2025-12-17T00:05:32Z",
  "file_sha256": "abc123...",
  "decompressed_sha256": "xyz789...",
  "compression": "zstd",
  "compression_level": 19
}
```

---

## Creating EPSS Bundles

### Prerequisites

**Build System Requirements**:
- Internet access (for fetching FIRST.org data)
- StellaOps Bundler CLI: `stellaops-bundler`
- ZSTD compression: `zstd` (v1.5+)
- Python 3.10+ (for verification scripts)

**Permissions**:
- Read access to FIRST.org EPSS API/CSV endpoints
- Write access to bundle staging directory
- (Optional) Signing key for DSSE signatures

### Daily Bundle Creation (Automated)

**Recommended Schedule**: Daily at 01:00 UTC (after FIRST publishes at ~00:00 UTC)

**Script**: `scripts/create-risk-bundle.sh`

```bash
#!/bin/bash
set -euo pipefail

BUNDLE_DATE=$(date -u +%Y-%m-%d)
BUNDLE_DIR="risk-bundle-${BUNDLE_DATE}"
STAGING_DIR="/tmp/stellaops-bundles/${BUNDLE_DIR}"

echo "Creating risk bundle for ${BUNDLE_DATE}..."

# 1. Create staging directory
mkdir -p "${STAGING_DIR}"/{epss,kev,advisories,signatures}

# 2. Fetch EPSS data from FIRST.org
echo "Fetching EPSS data..."
curl -sL "https://epss.empiricalsecurity.com/epss_scores-${BUNDLE_DATE}.csv.gz" \
  -o "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.gz"

# 3. Decompress and re-compress with ZSTD (better compression for offline)
gunzip "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.gz"
zstd -19 -q "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv" \
  -o "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.zst"
rm "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv"

# 4. Generate EPSS metadata
stellaops-bundler epss metadata \
  --file "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.zst" \
  --model-date "${BUNDLE_DATE}" \
  --output "${STAGING_DIR}/epss/epss_metadata.json"

# 5. Fetch KEV catalog
echo "Fetching KEV catalog..."
curl -sL "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json" \
  -o "${STAGING_DIR}/kev/kev-catalog.json"

# 6. Fetch advisory updates (optional, for comprehensive bundles)
# stellaops-bundler advisories fetch ...

# 7. Generate checksums
echo "Generating checksums..."
(cd "${STAGING_DIR}" && find . -type f ! -name "*.sha256sums" -exec sha256sum {} \;) \
  > "${STAGING_DIR}/signatures/bundle.sha256sums"

# 8. Generate manifest
stellaops-bundler manifest create \
  --bundle-dir "${STAGING_DIR}" \
  --bundle-id "${BUNDLE_DIR}" \
  --output "${STAGING_DIR}/manifest.json"

# 9. Sign bundle (if signing key available)
if [ -n "${SIGNING_KEY:-}" ]; then
  echo "Signing bundle..."
  stellaops-bundler sign \
    --manifest "${STAGING_DIR}/manifest.json" \
    --key "${SIGNING_KEY}" \
    --output "${STAGING_DIR}/signatures/bundle.dsse.json"
fi

# 10. Create tarball
echo "Creating tarball..."
tar -C "$(dirname "${STAGING_DIR}")" -czf "/var/stellaops/bundles/${BUNDLE_DIR}.tar.gz" \
  "$(basename "${STAGING_DIR}")"

echo "Bundle created: /var/stellaops/bundles/${BUNDLE_DIR}.tar.gz"
echo "Size: $(du -h /var/stellaops/bundles/${BUNDLE_DIR}.tar.gz | cut -f1)"

# 11. Verify bundle
stellaops-bundler verify "/var/stellaops/bundles/${BUNDLE_DIR}.tar.gz"
```

**Cron Schedule**:
```cron
# Daily at 01:00 UTC (after FIRST publishes EPSS at ~00:00 UTC)
0 1 * * * /opt/stellaops/scripts/create-risk-bundle.sh >> /var/log/stellaops/bundler.log 2>&1
```

---

## Distributing Bundles

### Transfer Methods

#### 1. Physical Media (Highest Security)

```bash
# Copy to USB drive
cp /var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz /media/usb/stellaops/

# Verify checksum
sha256sum /media/usb/stellaops/risk-bundle-2025-12-17.tar.gz
```

#### 2. Secure File Transfer (Network Isolation)

```bash
# SCP over dedicated management network
scp /var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz \
  admin@airgap-gateway.internal:/incoming/

# Verify after transfer
ssh admin@airgap-gateway.internal \
  "sha256sum /incoming/risk-bundle-2025-12-17.tar.gz"
```

#### 3. Offline Bundle Repository (CD/DVD)

```bash
# Burn to CD/DVD (for regulated industries)
growisofs -Z /dev/sr0 \
  -R -J -joliet-long \
  -V "StellaOps Risk Bundle 2025-12-17" \
  /var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz

# Verify disc
md5sum /dev/sr0 > risk-bundle-2025-12-17.md5
```

### Storage Recommendations

**Bundle Retention**:
- **Online bundler**: Keep last 90 days (rolling cleanup)
- **Air-gapped system**: Keep last 30 days minimum (for rollback)

**Naming Convention**:
- Pattern: `risk-bundle-YYYY-MM-DD.tar.gz`
- Example: `risk-bundle-2025-12-17.tar.gz`

**Directory Structure** (air-gapped system):
```
/opt/stellaops/bundles/
├── incoming/               # Transfer staging area
├── verified/               # Verified, ready to import
├── imported/               # Successfully imported (archive)
└── failed/                 # Failed verification/import (quarantine)
```

---

## Importing Bundles (Air-Gapped System)

### Pre-Import Verification

**Step 1: Transfer to Verified Directory**

```bash
# Transfer from incoming to verified (manual approval gate)
sudo mv /opt/stellaops/bundles/incoming/risk-bundle-2025-12-17.tar.gz \
        /opt/stellaops/bundles/verified/
```

**Step 2: Verify Bundle Integrity**

```bash
# Extract bundle
cd /opt/stellaops/bundles/verified
tar -xzf risk-bundle-2025-12-17.tar.gz

# Verify checksums
cd risk-bundle-2025-12-17
sha256sum -c signatures/bundle.sha256sums

# Expected output:
# epss/epss_scores-2025-12-17.csv.zst: OK
# epss/epss_metadata.json: OK
# kev/kev-catalog.json: OK
# manifest.json: OK
```

**Step 3: Verify DSSE Signature (if signed)**

```bash
stellaops-bundler verify-signature \
  --manifest manifest.json \
  --signature signatures/bundle.dsse.json \
  --trusted-keys /etc/stellaops/trusted-keys.json

# Expected output:
# ✓ Signature valid
# ✓ Key ID: stellaops-bundler-2025
# ✓ Signed at: 2025-12-17T01:05:00Z
```

### Import Procedure

**Step 4: Import Bundle**

```bash
# Import using stellaops CLI
stellaops offline import \
  --bundle /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
  --verify \
  --dry-run

# Review dry-run output, then execute
stellaops offline import \
  --bundle /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
  --verify
```

**Import Output**:
```
Importing risk bundle: risk-bundle-2025-12-17
  ✓ Manifest validated
  ✓ Checksums verified
  ✓ Signature verified

Importing EPSS data...
  Model Date: 2025-12-17
  Row Count: 231,417
  ✓ epss_import_runs created (import_run_id: 550e8400-...)
  ✓ epss_scores inserted (231,417 rows, 23.4s)
  ✓ epss_changes computed (12,345 changes, 8.1s)
  ✓ epss_current upserted (231,417 rows, 5.2s)
  ✓ Event emitted: epss.updated

Importing KEV catalog...
  Known Exploited Count: 1,247
  ✓ kev_catalog updated

Import completed successfully in 41.2s
```

**Step 5: Verify Import**

```bash
# Check EPSS status
stellaops epss status

# Expected output:
# EPSS Status:
#   Latest Model Date: 2025-12-17
#   Source: bundle://risk-bundle-2025-12-17
#   CVE Count: 231,417
#   Staleness: FRESH (0 days)
#   Import Time: 2025-12-17T10:30:00Z

# Query specific CVE to verify
stellaops epss get CVE-2024-12345

# Expected output:
# CVE-2024-12345
#   Score: 0.42357
#   Percentile: 88.2th
#   Model Date: 2025-12-17
#   Source: bundle://risk-bundle-2025-12-17
```

**Step 6: Archive Imported Bundle**

```bash
# Move to imported archive
sudo mv /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
        /opt/stellaops/bundles/imported/
```

---

## Automation (Air-Gapped System)

### Automated Import on Arrival

**Script**: `/opt/stellaops/scripts/auto-import-bundle.sh`

```bash
#!/bin/bash
set -euo pipefail

INCOMING_DIR="/opt/stellaops/bundles/incoming"
VERIFIED_DIR="/opt/stellaops/bundles/verified"
IMPORTED_DIR="/opt/stellaops/bundles/imported"
FAILED_DIR="/opt/stellaops/bundles/failed"
LOG_FILE="/var/log/stellaops/auto-import.log"

log() {
  echo "[$(date -Iseconds)] $*" | tee -a "${LOG_FILE}"
}

# Watch for new bundles in incoming/
for bundle in "${INCOMING_DIR}"/risk-bundle-*.tar.gz; do
  [ -f "${bundle}" ] || continue

  BUNDLE_NAME=$(basename "${bundle}")
  log "Detected new bundle: ${BUNDLE_NAME}"

  # Extract
  EXTRACT_DIR="${VERIFIED_DIR}/${BUNDLE_NAME%.tar.gz}"
  mkdir -p "${EXTRACT_DIR}"
  tar -xzf "${bundle}" -C "${VERIFIED_DIR}"

  # Verify checksums
  if ! (cd "${EXTRACT_DIR}" && sha256sum -c signatures/bundle.sha256sums > /dev/null 2>&1); then
    log "ERROR: Checksum verification failed for ${BUNDLE_NAME}"
    mv "${bundle}" "${FAILED_DIR}/"
    rm -rf "${EXTRACT_DIR}"
    continue
  fi

  log "Checksum verification passed"

  # Verify signature (if present)
  if [ -f "${EXTRACT_DIR}/signatures/bundle.dsse.json" ]; then
    if ! stellaops-bundler verify-signature \
      --manifest "${EXTRACT_DIR}/manifest.json" \
      --signature "${EXTRACT_DIR}/signatures/bundle.dsse.json" \
      --trusted-keys /etc/stellaops/trusted-keys.json > /dev/null 2>&1; then
      log "ERROR: Signature verification failed for ${BUNDLE_NAME}"
      mv "${bundle}" "${FAILED_DIR}/"
      rm -rf "${EXTRACT_DIR}"
      continue
    fi
    log "Signature verification passed"
  fi

  # Import
  if stellaops offline import --bundle "${bundle}" --verify >> "${LOG_FILE}" 2>&1; then
    log "Import successful for ${BUNDLE_NAME}"
    mv "${bundle}" "${IMPORTED_DIR}/"
    rm -rf "${EXTRACT_DIR}"
  else
    log "ERROR: Import failed for ${BUNDLE_NAME}"
    mv "${bundle}" "${FAILED_DIR}/"
  fi
done
```

**Systemd Service**: `/etc/systemd/system/stellaops-bundle-watcher.service`

```ini
[Unit]
Description=StellaOps Bundle Auto-Import Watcher
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/inotifywait -m -e close_write --format '%w%f' /opt/stellaops/bundles/incoming | \
          while read file; do /opt/stellaops/scripts/auto-import-bundle.sh; done
Restart=always
RestartSec=10
User=stellaops
Group=stellaops

[Install]
WantedBy=multi-user.target
```

**Enable Service**:
```bash
sudo systemctl enable stellaops-bundle-watcher
sudo systemctl start stellaops-bundle-watcher
```

---

## Staleness Handling

### Staleness Thresholds

| Days Since Model Date | Status | Action |
|-----------------------|--------|--------|
| 0-1 | FRESH | Normal operation |
| 2-7 | ACCEPTABLE | Continue, low-priority alert |
| 8-14 | STALE | Alert, plan bundle import |
| 15+ | VERY_STALE | Fallback to CVSS-only, urgent alert |

### Monitoring Staleness

**SQL Query**:
```sql
SELECT * FROM concelier.epss_model_staleness;

-- Output:
-- latest_model_date | latest_import_at       | days_stale | staleness_status
-- 2025-12-10        | 2025-12-10 10:30:00+00 | 7          | ACCEPTABLE
```

**Prometheus Metric**:
```promql
epss_model_staleness_days{instance="airgap-prod"}

# Alert rule:
- alert: EpssDataStale
  expr: epss_model_staleness_days > 7
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "EPSS data is stale ({{ $value }} days old)"
```

### Fallback Behavior

When EPSS data is VERY_STALE (>14 days):

**Automatic Fallback**:
- Scanner: Skip EPSS evidence, log warning
- Policy: Use CVSS-only scoring (no EPSS bonus)
- Notifications: Disabled EPSS-based alerts
- UI: Show staleness banner, disable EPSS filters

**Manual Override** (force continue using stale data):
```yaml
# etc/scanner.yaml
scanner:
  epss:
    staleness_policy: continue  # Options: fallback, continue, error
    max_staleness_days: 30      # Override 14-day default
```

---

## Troubleshooting

### Bundle Import Failed: Checksum Mismatch

**Symptom**:
```
ERROR: Checksum verification failed
epss/epss_scores-2025-12-17.csv.zst: FAILED
```

**Diagnosis**:
1. Verify bundle was not corrupted during transfer:
   ```bash
   # Compare with original
   sha256sum risk-bundle-2025-12-17.tar.gz
   ```

2. Re-transfer bundle from source

**Resolution**:
- Delete corrupted bundle: `rm risk-bundle-2025-12-17.tar.gz`
- Re-download/re-transfer from bundler system

### Bundle Import Failed: Signature Invalid

**Symptom**:
```
ERROR: Signature verification failed
Invalid signature or untrusted key
```

**Diagnosis**:
1. Check trusted keys configured:
   ```bash
   cat /etc/stellaops/trusted-keys.json
   ```

2. Verify key ID in bundle signature matches:
   ```bash
   jq '.signature.key_id' manifest.json
   ```

**Resolution**:
- Update trusted keys file with current bundler public key
- Or: Skip signature verification (if signatures optional):
  ```bash
  stellaops offline import --bundle risk-bundle-2025-12-17.tar.gz --skip-signature-verify
  ```

### No EPSS Data After Import

**Symptom**:
- Import succeeded, but `stellaops epss status` shows "No EPSS data"

**Diagnosis**:
```sql
-- Check import runs
SELECT * FROM concelier.epss_import_runs ORDER BY created_at DESC LIMIT 1;

-- Check epss_current count
SELECT COUNT(*) FROM concelier.epss_current;
```

**Resolution**:
1. If import_runs shows FAILED status:
   - Check error column: `SELECT error FROM concelier.epss_import_runs WHERE status = 'FAILED'`
   - Re-run import with verbose logging

2. If epss_current is empty:
   - Manually trigger upsert:
     ```sql
     -- Re-run upsert for latest model_date
     -- (This SQL is safe to re-run)
     INSERT INTO concelier.epss_current (cve_id, epss_score, percentile, model_date, import_run_id, updated_at)
     SELECT s.cve_id, s.epss_score, s.percentile, s.model_date, s.import_run_id, NOW()
     FROM concelier.epss_scores s
     WHERE s.model_date = (SELECT MAX(model_date) FROM concelier.epss_import_runs WHERE status = 'SUCCEEDED')
     ON CONFLICT (cve_id) DO UPDATE SET
       epss_score = EXCLUDED.epss_score,
       percentile = EXCLUDED.percentile,
       model_date = EXCLUDED.model_date,
       import_run_id = EXCLUDED.import_run_id,
       updated_at = NOW();
     ```

---

## Best Practices

### 1. Weekly Bundle Import Cadence

**Recommended Schedule**:
- **Minimum**: Weekly (every Monday)
- **Preferred**: Bi-weekly (Monday & Thursday)
- **Ideal**: Daily (if transfer logistics allow)

### 2. Bundle Verification Checklist

Before importing:
- [ ] Checksum verification passed
- [ ] Signature verification passed (if signed)
- [ ] Model date within acceptable staleness window
- [ ] Disk space available (estimate: 500MB per bundle)
- [ ] Backup current EPSS data (for rollback)

### 3. Rollback Plan

If new bundle causes issues:
```bash
# 1. Identify problematic import_run_id
SELECT import_run_id, model_date, status
FROM concelier.epss_import_runs
ORDER BY created_at DESC LIMIT 5;

# 2. Delete problematic import (cascades to epss_scores, epss_changes)
DELETE FROM concelier.epss_import_runs
WHERE import_run_id = '550e8400-...';

# 3. Restore epss_current from previous day
-- (Upsert from previous model_date as shown in troubleshooting)

# 4. Verify rollback
stellaops epss status
```

### 4. Audit Trail

Log all bundle imports for compliance:

**Audit Log Format** (`/var/log/stellaops/bundle-audit.log`):
```json
{
  "timestamp": "2025-12-17T10:30:00Z",
  "action": "import",
  "bundle_id": "risk-bundle-2025-12-17",
  "bundle_sha256": "abc123...",
  "imported_by": "admin@example.com",
  "import_run_id": "550e8400-e29b-41d4-a716-446655440000",
  "result": "SUCCESS",
  "row_count": 231417,
  "duration_seconds": 41.2
}
```

---

## Appendix: Bundle Creation Tools

### stellaops-bundler CLI Reference

```bash
# Create EPSS metadata
stellaops-bundler epss metadata \
  --file epss_scores-2025-12-17.csv.zst \
  --model-date 2025-12-17 \
  --output epss_metadata.json

# Create manifest
stellaops-bundler manifest create \
  --bundle-dir risk-bundle-2025-12-17 \
  --bundle-id risk-bundle-2025-12-17 \
  --output manifest.json

# Sign bundle
stellaops-bundler sign \
  --manifest manifest.json \
  --key /path/to/signing-key.pem \
  --output bundle.dsse.json

# Verify bundle
stellaops-bundler verify risk-bundle-2025-12-17.tar.gz
```

### Custom Bundle Scripts

Example for creating weekly bundles (7-day snapshots):

```bash
#!/bin/bash
# create-weekly-bundle.sh

WEEK_START=$(date -u -d "last monday" +%Y-%m-%d)
WEEK_END=$(date -u +%Y-%m-%d)
BUNDLE_ID="risk-bundle-weekly-${WEEK_START}"

echo "Creating weekly bundle: ${BUNDLE_ID}"

for day in $(seq 0 6); do
  CURRENT_DATE=$(date -u -d "${WEEK_START} + ${day} days" +%Y-%m-%d)
  # Fetch EPSS for each day...
  curl -sL "https://epss.empiricalsecurity.com/epss_scores-${CURRENT_DATE}.csv.gz" \
    -o "epss/epss_scores-${CURRENT_DATE}.csv.gz"
done

# Compress and bundle...
tar -czf "${BUNDLE_ID}.tar.gz" epss/ kev/ manifest.json
```

---

**Last Updated**: 2025-12-17
**Version**: 1.0
**Maintainer**: StellaOps Operations Team