Files

master 8bbfe4d2d2 feat(rate-limiting): Implement core rate limiting functionality with configuration, decision-making, metrics, middleware, and service registration

- Add RateLimitConfig for configuration management with YAML binding support.
- Introduce RateLimitDecision to encapsulate the result of rate limit checks.
- Implement RateLimitMetrics for OpenTelemetry metrics tracking.
- Create RateLimitMiddleware for enforcing rate limits on incoming requests.
- Develop RateLimitService to orchestrate instance and environment rate limit checks.
- Add RateLimitServiceCollectionExtensions for dependency injection registration.

2025-12-17 18:02:37 +02:00

19 KiB

Raw Blame History

EPSS Air-Gapped Bundles Guide

Overview

This guide describes how to create, distribute, and import EPSS (Exploit Prediction Scoring System) data bundles for air-gapped StellaOps deployments. EPSS bundles enable offline vulnerability risk scoring with the same probabilistic threat intelligence available to online deployments.

Key Concepts:

Risk Bundle: Aggregated security data (EPSS + KEV + advisories) for offline import
EPSS Snapshot: Single-day EPSS scores for all CVEs (~300k rows)
Staleness Threshold: How old EPSS data can be before fallback to CVSS-only
Deterministic Import: Same bundle imported twice yields identical database state

Bundle Structure

Standard Risk Bundle Layout

risk-bundle-2025-12-17/
├── manifest.json                           # Bundle metadata and checksums
├── epss/
│   ├── epss_scores-2025-12-17.csv.zst      # EPSS data (ZSTD compressed)
│   └── epss_metadata.json                  # EPSS provenance
├── kev/
│   └── kev-catalog.json                    # CISA KEV catalog
├── advisories/
│   ├── nvd-updates.ndjson.zst
│   └── ghsa-updates.ndjson.zst
└── signatures/
    ├── bundle.dsse.json                    # DSSE signature (optional)
    └── bundle.sha256sums                   # File integrity checksums

manifest.json

{
  "bundle_id": "risk-bundle-2025-12-17",
  "created_at": "2025-12-17T00:00:00Z",
  "created_by": "stellaops-bundler-v1.2.3",
  "bundle_type": "risk",
  "schema_version": "v1",
  "contents": {
    "epss": {
      "model_date": "2025-12-17",
      "file": "epss/epss_scores-2025-12-17.csv.zst",
      "sha256": "abc123...",
      "size_bytes": 15728640,
      "row_count": 231417
    },
    "kev": {
      "catalog_version": "2025-12-17",
      "file": "kev/kev-catalog.json",
      "sha256": "def456...",
      "known_exploited_count": 1247
    },
    "advisories": {
      "nvd": {
        "file": "advisories/nvd-updates.ndjson.zst",
        "sha256": "ghi789...",
        "record_count": 1523
      },
      "ghsa": {
        "file": "advisories/ghsa-updates.ndjson.zst",
        "sha256": "jkl012...",
        "record_count": 8734
      }
    }
  },
  "signature": {
    "type": "dsse",
    "file": "signatures/bundle.dsse.json",
    "key_id": "stellaops-bundler-2025",
    "algorithm": "ed25519"
  }
}

epss/epss_metadata.json

{
  "model_date": "2025-12-17",
  "model_version": "v2025.12.17",
  "published_date": "2025-12-17",
  "row_count": 231417,
  "source_uri": "https://epss.empiricalsecurity.com/epss_scores-2025-12-17.csv.gz",
  "retrieved_at": "2025-12-17T00:05:32Z",
  "file_sha256": "abc123...",
  "decompressed_sha256": "xyz789...",
  "compression": "zstd",
  "compression_level": 19
}

Creating EPSS Bundles

Prerequisites

Build System Requirements:

Internet access (for fetching FIRST.org data)
StellaOps Bundler CLI: stellaops-bundler
ZSTD compression: zstd (v1.5+)
Python 3.10+ (for verification scripts)

Permissions:

Read access to FIRST.org EPSS API/CSV endpoints
Write access to bundle staging directory
(Optional) Signing key for DSSE signatures

Daily Bundle Creation (Automated)

Recommended Schedule: Daily at 01:00 UTC (after FIRST publishes at ~00:00 UTC)

Script: scripts/create-risk-bundle.sh

#!/bin/bash
set -euo pipefail

BUNDLE_DATE=$(date -u +%Y-%m-%d)
BUNDLE_DIR="risk-bundle-${BUNDLE_DATE}"
STAGING_DIR="/tmp/stellaops-bundles/${BUNDLE_DIR}"

echo "Creating risk bundle for ${BUNDLE_DATE}..."

# 1. Create staging directory
mkdir -p "${STAGING_DIR}"/{epss,kev,advisories,signatures}

# 2. Fetch EPSS data from FIRST.org
echo "Fetching EPSS data..."
curl -sL "https://epss.empiricalsecurity.com/epss_scores-${BUNDLE_DATE}.csv.gz" \
  -o "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.gz"

# 3. Decompress and re-compress with ZSTD (better compression for offline)
gunzip "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.gz"
zstd -19 -q "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv" \
  -o "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.zst"
rm "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv"

# 4. Generate EPSS metadata
stellaops-bundler epss metadata \
  --file "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.zst" \
  --model-date "${BUNDLE_DATE}" \
  --output "${STAGING_DIR}/epss/epss_metadata.json"

# 5. Fetch KEV catalog
echo "Fetching KEV catalog..."
curl -sL "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json" \
  -o "${STAGING_DIR}/kev/kev-catalog.json"

# 6. Fetch advisory updates (optional, for comprehensive bundles)
# stellaops-bundler advisories fetch ...

# 7. Generate checksums
echo "Generating checksums..."
(cd "${STAGING_DIR}" && find . -type f ! -name "*.sha256sums" -exec sha256sum {} \;) \
  > "${STAGING_DIR}/signatures/bundle.sha256sums"

# 8. Generate manifest
stellaops-bundler manifest create \
  --bundle-dir "${STAGING_DIR}" \
  --bundle-id "${BUNDLE_DIR}" \
  --output "${STAGING_DIR}/manifest.json"

# 9. Sign bundle (if signing key available)
if [ -n "${SIGNING_KEY:-}" ]; then
  echo "Signing bundle..."
  stellaops-bundler sign \
    --manifest "${STAGING_DIR}/manifest.json" \
    --key "${SIGNING_KEY}" \
    --output "${STAGING_DIR}/signatures/bundle.dsse.json"
fi

# 10. Create tarball
echo "Creating tarball..."
tar -C "$(dirname "${STAGING_DIR}")" -czf "/var/stellaops/bundles/${BUNDLE_DIR}.tar.gz" \
  "$(basename "${STAGING_DIR}")"

echo "Bundle created: /var/stellaops/bundles/${BUNDLE_DIR}.tar.gz"
echo "Size: $(du -h /var/stellaops/bundles/${BUNDLE_DIR}.tar.gz | cut -f1)"

# 11. Verify bundle
stellaops-bundler verify "/var/stellaops/bundles/${BUNDLE_DIR}.tar.gz"

Cron Schedule:

# Daily at 01:00 UTC (after FIRST publishes EPSS at ~00:00 UTC)
0 1 * * * /opt/stellaops/scripts/create-risk-bundle.sh >> /var/log/stellaops/bundler.log 2>&1

Distributing Bundles

Transfer Methods

1. Physical Media (Highest Security)

# Copy to USB drive
cp /var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz /media/usb/stellaops/

# Verify checksum
sha256sum /media/usb/stellaops/risk-bundle-2025-12-17.tar.gz

2. Secure File Transfer (Network Isolation)

# SCP over dedicated management network
scp /var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz \
  admin@airgap-gateway.internal:/incoming/

# Verify after transfer
ssh admin@airgap-gateway.internal \
  "sha256sum /incoming/risk-bundle-2025-12-17.tar.gz"

3. Offline Bundle Repository (CD/DVD)

# Burn to CD/DVD (for regulated industries)
growisofs -Z /dev/sr0 \
  -R -J -joliet-long \
  -V "StellaOps Risk Bundle 2025-12-17" \
  /var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz

# Verify disc
md5sum /dev/sr0 > risk-bundle-2025-12-17.md5

Storage Recommendations

Bundle Retention:

Online bundler: Keep last 90 days (rolling cleanup)
Air-gapped system: Keep last 30 days minimum (for rollback)

Naming Convention:

Pattern: risk-bundle-YYYY-MM-DD.tar.gz
Example: risk-bundle-2025-12-17.tar.gz

Directory Structure (air-gapped system):

/opt/stellaops/bundles/
├── incoming/               # Transfer staging area
├── verified/               # Verified, ready to import
├── imported/               # Successfully imported (archive)
└── failed/                 # Failed verification/import (quarantine)

Importing Bundles (Air-Gapped System)

Pre-Import Verification

Step 1: Transfer to Verified Directory

# Transfer from incoming to verified (manual approval gate)
sudo mv /opt/stellaops/bundles/incoming/risk-bundle-2025-12-17.tar.gz \
        /opt/stellaops/bundles/verified/

Step 2: Verify Bundle Integrity

# Extract bundle
cd /opt/stellaops/bundles/verified
tar -xzf risk-bundle-2025-12-17.tar.gz

# Verify checksums
cd risk-bundle-2025-12-17
sha256sum -c signatures/bundle.sha256sums

# Expected output:
# epss/epss_scores-2025-12-17.csv.zst: OK
# epss/epss_metadata.json: OK
# kev/kev-catalog.json: OK
# manifest.json: OK

Step 3: Verify DSSE Signature (if signed)

stellaops-bundler verify-signature \
  --manifest manifest.json \
  --signature signatures/bundle.dsse.json \
  --trusted-keys /etc/stellaops/trusted-keys.json

# Expected output:
# ✓ Signature valid
# ✓ Key ID: stellaops-bundler-2025
# ✓ Signed at: 2025-12-17T01:05:00Z

Import Procedure

Step 4: Import Bundle

# Import using stellaops CLI
stellaops offline import \
  --bundle /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
  --verify \
  --dry-run

# Review dry-run output, then execute
stellaops offline import \
  --bundle /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
  --verify

Import Output:

Importing risk bundle: risk-bundle-2025-12-17
  ✓ Manifest validated
  ✓ Checksums verified
  ✓ Signature verified

Importing EPSS data...
  Model Date: 2025-12-17
  Row Count: 231,417
  ✓ epss_import_runs created (import_run_id: 550e8400-...)
  ✓ epss_scores inserted (231,417 rows, 23.4s)
  ✓ epss_changes computed (12,345 changes, 8.1s)
  ✓ epss_current upserted (231,417 rows, 5.2s)
  ✓ Event emitted: epss.updated

Importing KEV catalog...
  Known Exploited Count: 1,247
  ✓ kev_catalog updated

Import completed successfully in 41.2s

Step 5: Verify Import

# Check EPSS status
stellaops epss status

# Expected output:
# EPSS Status:
#   Latest Model Date: 2025-12-17
#   Source: bundle://risk-bundle-2025-12-17
#   CVE Count: 231,417
#   Staleness: FRESH (0 days)
#   Import Time: 2025-12-17T10:30:00Z

# Query specific CVE to verify
stellaops epss get CVE-2024-12345

# Expected output:
# CVE-2024-12345
#   Score: 0.42357
#   Percentile: 88.2th
#   Model Date: 2025-12-17
#   Source: bundle://risk-bundle-2025-12-17

Step 6: Archive Imported Bundle

# Move to imported archive
sudo mv /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
        /opt/stellaops/bundles/imported/

Automation (Air-Gapped System)

Automated Import on Arrival

Script: /opt/stellaops/scripts/auto-import-bundle.sh

#!/bin/bash
set -euo pipefail

INCOMING_DIR="/opt/stellaops/bundles/incoming"
VERIFIED_DIR="/opt/stellaops/bundles/verified"
IMPORTED_DIR="/opt/stellaops/bundles/imported"
FAILED_DIR="/opt/stellaops/bundles/failed"
LOG_FILE="/var/log/stellaops/auto-import.log"

log() {
  echo "[$(date -Iseconds)] $*" | tee -a "${LOG_FILE}"
}

# Watch for new bundles in incoming/
for bundle in "${INCOMING_DIR}"/risk-bundle-*.tar.gz; do
  [ -f "${bundle}" ] || continue

  BUNDLE_NAME=$(basename "${bundle}")
  log "Detected new bundle: ${BUNDLE_NAME}"

  # Extract
  EXTRACT_DIR="${VERIFIED_DIR}/${BUNDLE_NAME%.tar.gz}"
  mkdir -p "${EXTRACT_DIR}"
  tar -xzf "${bundle}" -C "${VERIFIED_DIR}"

  # Verify checksums
  if ! (cd "${EXTRACT_DIR}" && sha256sum -c signatures/bundle.sha256sums > /dev/null 2>&1); then
    log "ERROR: Checksum verification failed for ${BUNDLE_NAME}"
    mv "${bundle}" "${FAILED_DIR}/"
    rm -rf "${EXTRACT_DIR}"
    continue
  fi

  log "Checksum verification passed"

  # Verify signature (if present)
  if [ -f "${EXTRACT_DIR}/signatures/bundle.dsse.json" ]; then
    if ! stellaops-bundler verify-signature \
      --manifest "${EXTRACT_DIR}/manifest.json" \
      --signature "${EXTRACT_DIR}/signatures/bundle.dsse.json" \
      --trusted-keys /etc/stellaops/trusted-keys.json > /dev/null 2>&1; then
      log "ERROR: Signature verification failed for ${BUNDLE_NAME}"
      mv "${bundle}" "${FAILED_DIR}/"
      rm -rf "${EXTRACT_DIR}"
      continue
    fi
    log "Signature verification passed"
  fi

  # Import
  if stellaops offline import --bundle "${bundle}" --verify >> "${LOG_FILE}" 2>&1; then
    log "Import successful for ${BUNDLE_NAME}"
    mv "${bundle}" "${IMPORTED_DIR}/"
    rm -rf "${EXTRACT_DIR}"
  else
    log "ERROR: Import failed for ${BUNDLE_NAME}"
    mv "${bundle}" "${FAILED_DIR}/"
  fi
done

Systemd Service: /etc/systemd/system/stellaops-bundle-watcher.service

[Unit]
Description=StellaOps Bundle Auto-Import Watcher
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/inotifywait -m -e close_write --format '%w%f' /opt/stellaops/bundles/incoming | \
          while read file; do /opt/stellaops/scripts/auto-import-bundle.sh; done
Restart=always
RestartSec=10
User=stellaops
Group=stellaops

[Install]
WantedBy=multi-user.target

Enable Service:

sudo systemctl enable stellaops-bundle-watcher
sudo systemctl start stellaops-bundle-watcher

Staleness Handling

Staleness Thresholds

Days Since Model Date	Status	Action
0-1	FRESH	Normal operation
2-7	ACCEPTABLE	Continue, low-priority alert
8-14	STALE	Alert, plan bundle import
15+	VERY_STALE	Fallback to CVSS-only, urgent alert

Monitoring Staleness

SQL Query:

SELECT * FROM concelier.epss_model_staleness;

-- Output:
-- latest_model_date | latest_import_at       | days_stale | staleness_status
-- 2025-12-10        | 2025-12-10 10:30:00+00 | 7          | ACCEPTABLE

Prometheus Metric:

epss_model_staleness_days{instance="airgap-prod"}

# Alert rule:
- alert: EpssDataStale
  expr: epss_model_staleness_days > 7
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "EPSS data is stale ({{ $value }} days old)"

Fallback Behavior

When EPSS data is VERY_STALE (>14 days):

Automatic Fallback:

Scanner: Skip EPSS evidence, log warning
Policy: Use CVSS-only scoring (no EPSS bonus)
Notifications: Disabled EPSS-based alerts
UI: Show staleness banner, disable EPSS filters

Manual Override (force continue using stale data):

# etc/scanner.yaml
scanner:
  epss:
    staleness_policy: continue  # Options: fallback, continue, error
    max_staleness_days: 30      # Override 14-day default

Troubleshooting

Bundle Import Failed: Checksum Mismatch

Symptom:

ERROR: Checksum verification failed
epss/epss_scores-2025-12-17.csv.zst: FAILED

Diagnosis:

Verify bundle was not corrupted during transfer:

# Compare with original
sha256sum risk-bundle-2025-12-17.tar.gz

Re-transfer bundle from source

Resolution:

Delete corrupted bundle: rm risk-bundle-2025-12-17.tar.gz
Re-download/re-transfer from bundler system

Bundle Import Failed: Signature Invalid

Symptom:

ERROR: Signature verification failed
Invalid signature or untrusted key

Diagnosis:

Check trusted keys configured:
```
cat /etc/stellaops/trusted-keys.json
```
Verify key ID in bundle signature matches:
```
jq '.signature.key_id' manifest.json
```

Resolution:

Update trusted keys file with current bundler public key

Or: Skip signature verification (if signatures optional):

stellaops offline import --bundle risk-bundle-2025-12-17.tar.gz --skip-signature-verify

No EPSS Data After Import

Symptom:

Import succeeded, but stellaops epss status shows "No EPSS data"

Diagnosis:

-- Check import runs
SELECT * FROM concelier.epss_import_runs ORDER BY created_at DESC LIMIT 1;

-- Check epss_current count
SELECT COUNT(*) FROM concelier.epss_current;

Resolution:

If import_runs shows FAILED status:
- Check error column: SELECT error FROM concelier.epss_import_runs WHERE status = 'FAILED'
- Re-run import with verbose logging

If epss_current is empty:

Manually trigger upsert:

-- Re-run upsert for latest model_date
-- (This SQL is safe to re-run)
INSERT INTO concelier.epss_current (cve_id, epss_score, percentile, model_date, import_run_id, updated_at)
SELECT s.cve_id, s.epss_score, s.percentile, s.model_date, s.import_run_id, NOW()
FROM concelier.epss_scores s
WHERE s.model_date = (SELECT MAX(model_date) FROM concelier.epss_import_runs WHERE status = 'SUCCEEDED')
ON CONFLICT (cve_id) DO UPDATE SET
  epss_score = EXCLUDED.epss_score,
  percentile = EXCLUDED.percentile,
  model_date = EXCLUDED.model_date,
  import_run_id = EXCLUDED.import_run_id,
  updated_at = NOW();

Best Practices

1. Weekly Bundle Import Cadence

Recommended Schedule:

Minimum: Weekly (every Monday)
Preferred: Bi-weekly (Monday & Thursday)
Ideal: Daily (if transfer logistics allow)

2. Bundle Verification Checklist

Before importing:

Checksum verification passed
Signature verification passed (if signed)
Model date within acceptable staleness window
Disk space available (estimate: 500MB per bundle)
Backup current EPSS data (for rollback)

3. Rollback Plan

If new bundle causes issues:

# 1. Identify problematic import_run_id
SELECT import_run_id, model_date, status
FROM concelier.epss_import_runs
ORDER BY created_at DESC LIMIT 5;

# 2. Delete problematic import (cascades to epss_scores, epss_changes)
DELETE FROM concelier.epss_import_runs
WHERE import_run_id = '550e8400-...';

# 3. Restore epss_current from previous day
-- (Upsert from previous model_date as shown in troubleshooting)

# 4. Verify rollback
stellaops epss status

4. Audit Trail

Log all bundle imports for compliance:

Audit Log Format (/var/log/stellaops/bundle-audit.log):

{
  "timestamp": "2025-12-17T10:30:00Z",
  "action": "import",
  "bundle_id": "risk-bundle-2025-12-17",
  "bundle_sha256": "abc123...",
  "imported_by": "admin@example.com",
  "import_run_id": "550e8400-e29b-41d4-a716-446655440000",
  "result": "SUCCESS",
  "row_count": 231417,
  "duration_seconds": 41.2
}

Appendix: Bundle Creation Tools

stellaops-bundler CLI Reference

# Create EPSS metadata
stellaops-bundler epss metadata \
  --file epss_scores-2025-12-17.csv.zst \
  --model-date 2025-12-17 \
  --output epss_metadata.json

# Create manifest
stellaops-bundler manifest create \
  --bundle-dir risk-bundle-2025-12-17 \
  --bundle-id risk-bundle-2025-12-17 \
  --output manifest.json

# Sign bundle
stellaops-bundler sign \
  --manifest manifest.json \
  --key /path/to/signing-key.pem \
  --output bundle.dsse.json

# Verify bundle
stellaops-bundler verify risk-bundle-2025-12-17.tar.gz

Custom Bundle Scripts

Example for creating weekly bundles (7-day snapshots):

#!/bin/bash
# create-weekly-bundle.sh

WEEK_START=$(date -u -d "last monday" +%Y-%m-%d)
WEEK_END=$(date -u +%Y-%m-%d)
BUNDLE_ID="risk-bundle-weekly-${WEEK_START}"

echo "Creating weekly bundle: ${BUNDLE_ID}"

for day in $(seq 0 6); do
  CURRENT_DATE=$(date -u -d "${WEEK_START} + ${day} days" +%Y-%m-%d)
  # Fetch EPSS for each day...
  curl -sL "https://epss.empiricalsecurity.com/epss_scores-${CURRENT_DATE}.csv.gz" \
    -o "epss/epss_scores-${CURRENT_DATE}.csv.gz"
done

# Compress and bundle...
tar -czf "${BUNDLE_ID}.tar.gz" epss/ kev/ manifest.json

Last Updated: 2025-12-17 Version: 1.0 Maintainer: StellaOps Operations Team

19 KiB Raw Blame History

EPSS Air-Gapped Bundles Guide

Overview

Bundle Structure

Standard Risk Bundle Layout

manifest.json

epss/epss_metadata.json

Creating EPSS Bundles

Prerequisites

Daily Bundle Creation (Automated)

Distributing Bundles

Transfer Methods

1. Physical Media (Highest Security)

2. Secure File Transfer (Network Isolation)

3. Offline Bundle Repository (CD/DVD)

Storage Recommendations

Importing Bundles (Air-Gapped System)

Pre-Import Verification

Import Procedure

Automation (Air-Gapped System)

Automated Import on Arrival

Staleness Handling

Staleness Thresholds

Monitoring Staleness

Fallback Behavior

Troubleshooting

Bundle Import Failed: Checksum Mismatch

Bundle Import Failed: Signature Invalid

No EPSS Data After Import

Best Practices

1. Weekly Bundle Import Cadence

2. Bundle Verification Checklist

3. Rollback Plan

4. Audit Trail

Appendix: Bundle Creation Tools

stellaops-bundler CLI Reference

Custom Bundle Scripts

19 KiB

Raw Blame History