feat(rate-limiting): Implement core rate limiting functionality with configuration, decision-making, metrics, middleware, and service registration
- Add RateLimitConfig for configuration management with YAML binding support. - Introduce RateLimitDecision to encapsulate the result of rate limit checks. - Implement RateLimitMetrics for OpenTelemetry metrics tracking. - Create RateLimitMiddleware for enforcing rate limits on incoming requests. - Develop RateLimitService to orchestrate instance and environment rate limit checks. - Add RateLimitServiceCollectionExtensions for dependency injection registration.
This commit is contained in:
master
732
docs/airgap/epss-bundles.md
Normal file
732
docs/airgap/epss-bundles.md
Normal file
@@ -0,0 +1,732 @@
|
||||
# EPSS Air-Gapped Bundles Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This guide describes how to create, distribute, and import EPSS (Exploit Prediction Scoring System) data bundles for air-gapped StellaOps deployments. EPSS bundles enable offline vulnerability risk scoring with the same probabilistic threat intelligence available to online deployments.
|
||||
|
||||
**Key Concepts**:
|
||||
- **Risk Bundle**: Aggregated security data (EPSS + KEV + advisories) for offline import
|
||||
- **EPSS Snapshot**: Single-day EPSS scores for all CVEs (~300k rows)
|
||||
- **Staleness Threshold**: How old EPSS data can be before fallback to CVSS-only
|
||||
- **Deterministic Import**: Same bundle imported twice yields identical database state
|
||||
|
||||
---
|
||||
|
||||
## Bundle Structure
|
||||
|
||||
### Standard Risk Bundle Layout
|
||||
|
||||
```
|
||||
risk-bundle-2025-12-17/
|
||||
├── manifest.json # Bundle metadata and checksums
|
||||
├── epss/
|
||||
│ ├── epss_scores-2025-12-17.csv.zst # EPSS data (ZSTD compressed)
|
||||
│ └── epss_metadata.json # EPSS provenance
|
||||
├── kev/
|
||||
│ └── kev-catalog.json # CISA KEV catalog
|
||||
├── advisories/
|
||||
│ ├── nvd-updates.ndjson.zst
|
||||
│ └── ghsa-updates.ndjson.zst
|
||||
└── signatures/
|
||||
├── bundle.dsse.json # DSSE signature (optional)
|
||||
└── bundle.sha256sums # File integrity checksums
|
||||
```
|
||||
|
||||
### manifest.json
|
||||
|
||||
```json
|
||||
{
|
||||
"bundle_id": "risk-bundle-2025-12-17",
|
||||
"created_at": "2025-12-17T00:00:00Z",
|
||||
"created_by": "stellaops-bundler-v1.2.3",
|
||||
"bundle_type": "risk",
|
||||
"schema_version": "v1",
|
||||
"contents": {
|
||||
"epss": {
|
||||
"model_date": "2025-12-17",
|
||||
"file": "epss/epss_scores-2025-12-17.csv.zst",
|
||||
"sha256": "abc123...",
|
||||
"size_bytes": 15728640,
|
||||
"row_count": 231417
|
||||
},
|
||||
"kev": {
|
||||
"catalog_version": "2025-12-17",
|
||||
"file": "kev/kev-catalog.json",
|
||||
"sha256": "def456...",
|
||||
"known_exploited_count": 1247
|
||||
},
|
||||
"advisories": {
|
||||
"nvd": {
|
||||
"file": "advisories/nvd-updates.ndjson.zst",
|
||||
"sha256": "ghi789...",
|
||||
"record_count": 1523
|
||||
},
|
||||
"ghsa": {
|
||||
"file": "advisories/ghsa-updates.ndjson.zst",
|
||||
"sha256": "jkl012...",
|
||||
"record_count": 8734
|
||||
}
|
||||
}
|
||||
},
|
||||
"signature": {
|
||||
"type": "dsse",
|
||||
"file": "signatures/bundle.dsse.json",
|
||||
"key_id": "stellaops-bundler-2025",
|
||||
"algorithm": "ed25519"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### epss/epss_metadata.json
|
||||
|
||||
```json
|
||||
{
|
||||
"model_date": "2025-12-17",
|
||||
"model_version": "v2025.12.17",
|
||||
"published_date": "2025-12-17",
|
||||
"row_count": 231417,
|
||||
"source_uri": "https://epss.empiricalsecurity.com/epss_scores-2025-12-17.csv.gz",
|
||||
"retrieved_at": "2025-12-17T00:05:32Z",
|
||||
"file_sha256": "abc123...",
|
||||
"decompressed_sha256": "xyz789...",
|
||||
"compression": "zstd",
|
||||
"compression_level": 19
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Creating EPSS Bundles
|
||||
|
||||
### Prerequisites
|
||||
|
||||
**Build System Requirements**:
|
||||
- Internet access (for fetching FIRST.org data)
|
||||
- StellaOps Bundler CLI: `stellaops-bundler`
|
||||
- ZSTD compression: `zstd` (v1.5+)
|
||||
- Python 3.10+ (for verification scripts)
|
||||
|
||||
**Permissions**:
|
||||
- Read access to FIRST.org EPSS API/CSV endpoints
|
||||
- Write access to bundle staging directory
|
||||
- (Optional) Signing key for DSSE signatures
|
||||
|
||||
### Daily Bundle Creation (Automated)
|
||||
|
||||
**Recommended Schedule**: Daily at 01:00 UTC (after FIRST publishes at ~00:00 UTC)
|
||||
|
||||
**Script**: `scripts/create-risk-bundle.sh`
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
BUNDLE_DATE=$(date -u +%Y-%m-%d)
|
||||
BUNDLE_DIR="risk-bundle-${BUNDLE_DATE}"
|
||||
STAGING_DIR="/tmp/stellaops-bundles/${BUNDLE_DIR}"
|
||||
|
||||
echo "Creating risk bundle for ${BUNDLE_DATE}..."
|
||||
|
||||
# 1. Create staging directory
|
||||
mkdir -p "${STAGING_DIR}"/{epss,kev,advisories,signatures}
|
||||
|
||||
# 2. Fetch EPSS data from FIRST.org
|
||||
echo "Fetching EPSS data..."
|
||||
curl -sL "https://epss.empiricalsecurity.com/epss_scores-${BUNDLE_DATE}.csv.gz" \
|
||||
-o "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.gz"
|
||||
|
||||
# 3. Decompress and re-compress with ZSTD (better compression for offline)
|
||||
gunzip "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.gz"
|
||||
zstd -19 -q "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv" \
|
||||
-o "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.zst"
|
||||
rm "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv"
|
||||
|
||||
# 4. Generate EPSS metadata
|
||||
stellaops-bundler epss metadata \
|
||||
--file "${STAGING_DIR}/epss/epss_scores-${BUNDLE_DATE}.csv.zst" \
|
||||
--model-date "${BUNDLE_DATE}" \
|
||||
--output "${STAGING_DIR}/epss/epss_metadata.json"
|
||||
|
||||
# 5. Fetch KEV catalog
|
||||
echo "Fetching KEV catalog..."
|
||||
curl -sL "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json" \
|
||||
-o "${STAGING_DIR}/kev/kev-catalog.json"
|
||||
|
||||
# 6. Fetch advisory updates (optional, for comprehensive bundles)
|
||||
# stellaops-bundler advisories fetch ...
|
||||
|
||||
# 7. Generate checksums
|
||||
echo "Generating checksums..."
|
||||
(cd "${STAGING_DIR}" && find . -type f ! -name "*.sha256sums" -exec sha256sum {} \;) \
|
||||
> "${STAGING_DIR}/signatures/bundle.sha256sums"
|
||||
|
||||
# 8. Generate manifest
|
||||
stellaops-bundler manifest create \
|
||||
--bundle-dir "${STAGING_DIR}" \
|
||||
--bundle-id "${BUNDLE_DIR}" \
|
||||
--output "${STAGING_DIR}/manifest.json"
|
||||
|
||||
# 9. Sign bundle (if signing key available)
|
||||
if [ -n "${SIGNING_KEY:-}" ]; then
|
||||
echo "Signing bundle..."
|
||||
stellaops-bundler sign \
|
||||
--manifest "${STAGING_DIR}/manifest.json" \
|
||||
--key "${SIGNING_KEY}" \
|
||||
--output "${STAGING_DIR}/signatures/bundle.dsse.json"
|
||||
fi
|
||||
|
||||
# 10. Create tarball
|
||||
echo "Creating tarball..."
|
||||
tar -C "$(dirname "${STAGING_DIR}")" -czf "/var/stellaops/bundles/${BUNDLE_DIR}.tar.gz" \
|
||||
"$(basename "${STAGING_DIR}")"
|
||||
|
||||
echo "Bundle created: /var/stellaops/bundles/${BUNDLE_DIR}.tar.gz"
|
||||
echo "Size: $(du -h /var/stellaops/bundles/${BUNDLE_DIR}.tar.gz | cut -f1)"
|
||||
|
||||
# 11. Verify bundle
|
||||
stellaops-bundler verify "/var/stellaops/bundles/${BUNDLE_DIR}.tar.gz"
|
||||
```
|
||||
|
||||
**Cron Schedule**:
|
||||
```cron
|
||||
# Daily at 01:00 UTC (after FIRST publishes EPSS at ~00:00 UTC)
|
||||
0 1 * * * /opt/stellaops/scripts/create-risk-bundle.sh >> /var/log/stellaops/bundler.log 2>&1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Distributing Bundles
|
||||
|
||||
### Transfer Methods
|
||||
|
||||
#### 1. Physical Media (Highest Security)
|
||||
|
||||
```bash
|
||||
# Copy to USB drive
|
||||
cp /var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz /media/usb/stellaops/
|
||||
|
||||
# Verify checksum
|
||||
sha256sum /media/usb/stellaops/risk-bundle-2025-12-17.tar.gz
|
||||
```
|
||||
|
||||
#### 2. Secure File Transfer (Network Isolation)
|
||||
|
||||
```bash
|
||||
# SCP over dedicated management network
|
||||
scp /var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz \
|
||||
admin@airgap-gateway.internal:/incoming/
|
||||
|
||||
# Verify after transfer
|
||||
ssh admin@airgap-gateway.internal \
|
||||
"sha256sum /incoming/risk-bundle-2025-12-17.tar.gz"
|
||||
```
|
||||
|
||||
#### 3. Offline Bundle Repository (CD/DVD)
|
||||
|
||||
```bash
|
||||
# Burn to CD/DVD (for regulated industries)
|
||||
growisofs -Z /dev/sr0 \
|
||||
-R -J -joliet-long \
|
||||
-V "StellaOps Risk Bundle 2025-12-17" \
|
||||
/var/stellaops/bundles/risk-bundle-2025-12-17.tar.gz
|
||||
|
||||
# Verify disc
|
||||
md5sum /dev/sr0 > risk-bundle-2025-12-17.md5
|
||||
```
|
||||
|
||||
### Storage Recommendations
|
||||
|
||||
**Bundle Retention**:
|
||||
- **Online bundler**: Keep last 90 days (rolling cleanup)
|
||||
- **Air-gapped system**: Keep last 30 days minimum (for rollback)
|
||||
|
||||
**Naming Convention**:
|
||||
- Pattern: `risk-bundle-YYYY-MM-DD.tar.gz`
|
||||
- Example: `risk-bundle-2025-12-17.tar.gz`
|
||||
|
||||
**Directory Structure** (air-gapped system):
|
||||
```
|
||||
/opt/stellaops/bundles/
|
||||
├── incoming/ # Transfer staging area
|
||||
├── verified/ # Verified, ready to import
|
||||
├── imported/ # Successfully imported (archive)
|
||||
└── failed/ # Failed verification/import (quarantine)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Importing Bundles (Air-Gapped System)
|
||||
|
||||
### Pre-Import Verification
|
||||
|
||||
**Step 1: Transfer to Verified Directory**
|
||||
|
||||
```bash
|
||||
# Transfer from incoming to verified (manual approval gate)
|
||||
sudo mv /opt/stellaops/bundles/incoming/risk-bundle-2025-12-17.tar.gz \
|
||||
/opt/stellaops/bundles/verified/
|
||||
```
|
||||
|
||||
**Step 2: Verify Bundle Integrity**
|
||||
|
||||
```bash
|
||||
# Extract bundle
|
||||
cd /opt/stellaops/bundles/verified
|
||||
tar -xzf risk-bundle-2025-12-17.tar.gz
|
||||
|
||||
# Verify checksums
|
||||
cd risk-bundle-2025-12-17
|
||||
sha256sum -c signatures/bundle.sha256sums
|
||||
|
||||
# Expected output:
|
||||
# epss/epss_scores-2025-12-17.csv.zst: OK
|
||||
# epss/epss_metadata.json: OK
|
||||
# kev/kev-catalog.json: OK
|
||||
# manifest.json: OK
|
||||
```
|
||||
|
||||
**Step 3: Verify DSSE Signature (if signed)**
|
||||
|
||||
```bash
|
||||
stellaops-bundler verify-signature \
|
||||
--manifest manifest.json \
|
||||
--signature signatures/bundle.dsse.json \
|
||||
--trusted-keys /etc/stellaops/trusted-keys.json
|
||||
|
||||
# Expected output:
|
||||
# ✓ Signature valid
|
||||
# ✓ Key ID: stellaops-bundler-2025
|
||||
# ✓ Signed at: 2025-12-17T01:05:00Z
|
||||
```
|
||||
|
||||
### Import Procedure
|
||||
|
||||
**Step 4: Import Bundle**
|
||||
|
||||
```bash
|
||||
# Import using stellaops CLI
|
||||
stellaops offline import \
|
||||
--bundle /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
|
||||
--verify \
|
||||
--dry-run
|
||||
|
||||
# Review dry-run output, then execute
|
||||
stellaops offline import \
|
||||
--bundle /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
|
||||
--verify
|
||||
```
|
||||
|
||||
**Import Output**:
|
||||
```
|
||||
Importing risk bundle: risk-bundle-2025-12-17
|
||||
✓ Manifest validated
|
||||
✓ Checksums verified
|
||||
✓ Signature verified
|
||||
|
||||
Importing EPSS data...
|
||||
Model Date: 2025-12-17
|
||||
Row Count: 231,417
|
||||
✓ epss_import_runs created (import_run_id: 550e8400-...)
|
||||
✓ epss_scores inserted (231,417 rows, 23.4s)
|
||||
✓ epss_changes computed (12,345 changes, 8.1s)
|
||||
✓ epss_current upserted (231,417 rows, 5.2s)
|
||||
✓ Event emitted: epss.updated
|
||||
|
||||
Importing KEV catalog...
|
||||
Known Exploited Count: 1,247
|
||||
✓ kev_catalog updated
|
||||
|
||||
Import completed successfully in 41.2s
|
||||
```
|
||||
|
||||
**Step 5: Verify Import**
|
||||
|
||||
```bash
|
||||
# Check EPSS status
|
||||
stellaops epss status
|
||||
|
||||
# Expected output:
|
||||
# EPSS Status:
|
||||
# Latest Model Date: 2025-12-17
|
||||
# Source: bundle://risk-bundle-2025-12-17
|
||||
# CVE Count: 231,417
|
||||
# Staleness: FRESH (0 days)
|
||||
# Import Time: 2025-12-17T10:30:00Z
|
||||
|
||||
# Query specific CVE to verify
|
||||
stellaops epss get CVE-2024-12345
|
||||
|
||||
# Expected output:
|
||||
# CVE-2024-12345
|
||||
# Score: 0.42357
|
||||
# Percentile: 88.2th
|
||||
# Model Date: 2025-12-17
|
||||
# Source: bundle://risk-bundle-2025-12-17
|
||||
```
|
||||
|
||||
**Step 6: Archive Imported Bundle**
|
||||
|
||||
```bash
|
||||
# Move to imported archive
|
||||
sudo mv /opt/stellaops/bundles/verified/risk-bundle-2025-12-17.tar.gz \
|
||||
/opt/stellaops/bundles/imported/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Automation (Air-Gapped System)
|
||||
|
||||
### Automated Import on Arrival
|
||||
|
||||
**Script**: `/opt/stellaops/scripts/auto-import-bundle.sh`
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
INCOMING_DIR="/opt/stellaops/bundles/incoming"
|
||||
VERIFIED_DIR="/opt/stellaops/bundles/verified"
|
||||
IMPORTED_DIR="/opt/stellaops/bundles/imported"
|
||||
FAILED_DIR="/opt/stellaops/bundles/failed"
|
||||
LOG_FILE="/var/log/stellaops/auto-import.log"
|
||||
|
||||
log() {
|
||||
echo "[$(date -Iseconds)] $*" | tee -a "${LOG_FILE}"
|
||||
}
|
||||
|
||||
# Watch for new bundles in incoming/
|
||||
for bundle in "${INCOMING_DIR}"/risk-bundle-*.tar.gz; do
|
||||
[ -f "${bundle}" ] || continue
|
||||
|
||||
BUNDLE_NAME=$(basename "${bundle}")
|
||||
log "Detected new bundle: ${BUNDLE_NAME}"
|
||||
|
||||
# Extract
|
||||
EXTRACT_DIR="${VERIFIED_DIR}/${BUNDLE_NAME%.tar.gz}"
|
||||
mkdir -p "${EXTRACT_DIR}"
|
||||
tar -xzf "${bundle}" -C "${VERIFIED_DIR}"
|
||||
|
||||
# Verify checksums
|
||||
if ! (cd "${EXTRACT_DIR}" && sha256sum -c signatures/bundle.sha256sums > /dev/null 2>&1); then
|
||||
log "ERROR: Checksum verification failed for ${BUNDLE_NAME}"
|
||||
mv "${bundle}" "${FAILED_DIR}/"
|
||||
rm -rf "${EXTRACT_DIR}"
|
||||
continue
|
||||
fi
|
||||
|
||||
log "Checksum verification passed"
|
||||
|
||||
# Verify signature (if present)
|
||||
if [ -f "${EXTRACT_DIR}/signatures/bundle.dsse.json" ]; then
|
||||
if ! stellaops-bundler verify-signature \
|
||||
--manifest "${EXTRACT_DIR}/manifest.json" \
|
||||
--signature "${EXTRACT_DIR}/signatures/bundle.dsse.json" \
|
||||
--trusted-keys /etc/stellaops/trusted-keys.json > /dev/null 2>&1; then
|
||||
log "ERROR: Signature verification failed for ${BUNDLE_NAME}"
|
||||
mv "${bundle}" "${FAILED_DIR}/"
|
||||
rm -rf "${EXTRACT_DIR}"
|
||||
continue
|
||||
fi
|
||||
log "Signature verification passed"
|
||||
fi
|
||||
|
||||
# Import
|
||||
if stellaops offline import --bundle "${bundle}" --verify >> "${LOG_FILE}" 2>&1; then
|
||||
log "Import successful for ${BUNDLE_NAME}"
|
||||
mv "${bundle}" "${IMPORTED_DIR}/"
|
||||
rm -rf "${EXTRACT_DIR}"
|
||||
else
|
||||
log "ERROR: Import failed for ${BUNDLE_NAME}"
|
||||
mv "${bundle}" "${FAILED_DIR}/"
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
**Systemd Service**: `/etc/systemd/system/stellaops-bundle-watcher.service`
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=StellaOps Bundle Auto-Import Watcher
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/inotifywait -m -e close_write --format '%w%f' /opt/stellaops/bundles/incoming | \
|
||||
while read file; do /opt/stellaops/scripts/auto-import-bundle.sh; done
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
User=stellaops
|
||||
Group=stellaops
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
**Enable Service**:
|
||||
```bash
|
||||
sudo systemctl enable stellaops-bundle-watcher
|
||||
sudo systemctl start stellaops-bundle-watcher
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Staleness Handling
|
||||
|
||||
### Staleness Thresholds
|
||||
|
||||
| Days Since Model Date | Status | Action |
|
||||
|-----------------------|--------|--------|
|
||||
| 0-1 | FRESH | Normal operation |
|
||||
| 2-7 | ACCEPTABLE | Continue, low-priority alert |
|
||||
| 8-14 | STALE | Alert, plan bundle import |
|
||||
| 15+ | VERY_STALE | Fallback to CVSS-only, urgent alert |
|
||||
|
||||
### Monitoring Staleness
|
||||
|
||||
**SQL Query**:
|
||||
```sql
|
||||
SELECT * FROM concelier.epss_model_staleness;
|
||||
|
||||
-- Output:
|
||||
-- latest_model_date | latest_import_at | days_stale | staleness_status
|
||||
-- 2025-12-10 | 2025-12-10 10:30:00+00 | 7 | ACCEPTABLE
|
||||
```
|
||||
|
||||
**Prometheus Metric**:
|
||||
```promql
|
||||
epss_model_staleness_days{instance="airgap-prod"}
|
||||
|
||||
# Alert rule:
|
||||
- alert: EpssDataStale
|
||||
expr: epss_model_staleness_days > 7
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "EPSS data is stale ({{ $value }} days old)"
|
||||
```
|
||||
|
||||
### Fallback Behavior
|
||||
|
||||
When EPSS data is VERY_STALE (>14 days):
|
||||
|
||||
**Automatic Fallback**:
|
||||
- Scanner: Skip EPSS evidence, log warning
|
||||
- Policy: Use CVSS-only scoring (no EPSS bonus)
|
||||
- Notifications: Disabled EPSS-based alerts
|
||||
- UI: Show staleness banner, disable EPSS filters
|
||||
|
||||
**Manual Override** (force continue using stale data):
|
||||
```yaml
|
||||
# etc/scanner.yaml
|
||||
scanner:
|
||||
epss:
|
||||
staleness_policy: continue # Options: fallback, continue, error
|
||||
max_staleness_days: 30 # Override 14-day default
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Bundle Import Failed: Checksum Mismatch
|
||||
|
||||
**Symptom**:
|
||||
```
|
||||
ERROR: Checksum verification failed
|
||||
epss/epss_scores-2025-12-17.csv.zst: FAILED
|
||||
```
|
||||
|
||||
**Diagnosis**:
|
||||
1. Verify bundle was not corrupted during transfer:
|
||||
```bash
|
||||
# Compare with original
|
||||
sha256sum risk-bundle-2025-12-17.tar.gz
|
||||
```
|
||||
|
||||
2. Re-transfer bundle from source
|
||||
|
||||
**Resolution**:
|
||||
- Delete corrupted bundle: `rm risk-bundle-2025-12-17.tar.gz`
|
||||
- Re-download/re-transfer from bundler system
|
||||
|
||||
### Bundle Import Failed: Signature Invalid
|
||||
|
||||
**Symptom**:
|
||||
```
|
||||
ERROR: Signature verification failed
|
||||
Invalid signature or untrusted key
|
||||
```
|
||||
|
||||
**Diagnosis**:
|
||||
1. Check trusted keys configured:
|
||||
```bash
|
||||
cat /etc/stellaops/trusted-keys.json
|
||||
```
|
||||
|
||||
2. Verify key ID in bundle signature matches:
|
||||
```bash
|
||||
jq '.signature.key_id' manifest.json
|
||||
```
|
||||
|
||||
**Resolution**:
|
||||
- Update trusted keys file with current bundler public key
|
||||
- Or: Skip signature verification (if signatures optional):
|
||||
```bash
|
||||
stellaops offline import --bundle risk-bundle-2025-12-17.tar.gz --skip-signature-verify
|
||||
```
|
||||
|
||||
### No EPSS Data After Import
|
||||
|
||||
**Symptom**:
|
||||
- Import succeeded, but `stellaops epss status` shows "No EPSS data"
|
||||
|
||||
**Diagnosis**:
|
||||
```sql
|
||||
-- Check import runs
|
||||
SELECT * FROM concelier.epss_import_runs ORDER BY created_at DESC LIMIT 1;
|
||||
|
||||
-- Check epss_current count
|
||||
SELECT COUNT(*) FROM concelier.epss_current;
|
||||
```
|
||||
|
||||
**Resolution**:
|
||||
1. If import_runs shows FAILED status:
|
||||
- Check error column: `SELECT error FROM concelier.epss_import_runs WHERE status = 'FAILED'`
|
||||
- Re-run import with verbose logging
|
||||
|
||||
2. If epss_current is empty:
|
||||
- Manually trigger upsert:
|
||||
```sql
|
||||
-- Re-run upsert for latest model_date
|
||||
-- (This SQL is safe to re-run)
|
||||
INSERT INTO concelier.epss_current (cve_id, epss_score, percentile, model_date, import_run_id, updated_at)
|
||||
SELECT s.cve_id, s.epss_score, s.percentile, s.model_date, s.import_run_id, NOW()
|
||||
FROM concelier.epss_scores s
|
||||
WHERE s.model_date = (SELECT MAX(model_date) FROM concelier.epss_import_runs WHERE status = 'SUCCEEDED')
|
||||
ON CONFLICT (cve_id) DO UPDATE SET
|
||||
epss_score = EXCLUDED.epss_score,
|
||||
percentile = EXCLUDED.percentile,
|
||||
model_date = EXCLUDED.model_date,
|
||||
import_run_id = EXCLUDED.import_run_id,
|
||||
updated_at = NOW();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Weekly Bundle Import Cadence
|
||||
|
||||
**Recommended Schedule**:
|
||||
- **Minimum**: Weekly (every Monday)
|
||||
- **Preferred**: Bi-weekly (Monday & Thursday)
|
||||
- **Ideal**: Daily (if transfer logistics allow)
|
||||
|
||||
### 2. Bundle Verification Checklist
|
||||
|
||||
Before importing:
|
||||
- [ ] Checksum verification passed
|
||||
- [ ] Signature verification passed (if signed)
|
||||
- [ ] Model date within acceptable staleness window
|
||||
- [ ] Disk space available (estimate: 500MB per bundle)
|
||||
- [ ] Backup current EPSS data (for rollback)
|
||||
|
||||
### 3. Rollback Plan
|
||||
|
||||
If new bundle causes issues:
|
||||
```bash
|
||||
# 1. Identify problematic import_run_id
|
||||
SELECT import_run_id, model_date, status
|
||||
FROM concelier.epss_import_runs
|
||||
ORDER BY created_at DESC LIMIT 5;
|
||||
|
||||
# 2. Delete problematic import (cascades to epss_scores, epss_changes)
|
||||
DELETE FROM concelier.epss_import_runs
|
||||
WHERE import_run_id = '550e8400-...';
|
||||
|
||||
# 3. Restore epss_current from previous day
|
||||
-- (Upsert from previous model_date as shown in troubleshooting)
|
||||
|
||||
# 4. Verify rollback
|
||||
stellaops epss status
|
||||
```
|
||||
|
||||
### 4. Audit Trail
|
||||
|
||||
Log all bundle imports for compliance:
|
||||
|
||||
**Audit Log Format** (`/var/log/stellaops/bundle-audit.log`):
|
||||
```json
|
||||
{
|
||||
"timestamp": "2025-12-17T10:30:00Z",
|
||||
"action": "import",
|
||||
"bundle_id": "risk-bundle-2025-12-17",
|
||||
"bundle_sha256": "abc123...",
|
||||
"imported_by": "admin@example.com",
|
||||
"import_run_id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"result": "SUCCESS",
|
||||
"row_count": 231417,
|
||||
"duration_seconds": 41.2
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Bundle Creation Tools
|
||||
|
||||
### stellaops-bundler CLI Reference
|
||||
|
||||
```bash
|
||||
# Create EPSS metadata
|
||||
stellaops-bundler epss metadata \
|
||||
--file epss_scores-2025-12-17.csv.zst \
|
||||
--model-date 2025-12-17 \
|
||||
--output epss_metadata.json
|
||||
|
||||
# Create manifest
|
||||
stellaops-bundler manifest create \
|
||||
--bundle-dir risk-bundle-2025-12-17 \
|
||||
--bundle-id risk-bundle-2025-12-17 \
|
||||
--output manifest.json
|
||||
|
||||
# Sign bundle
|
||||
stellaops-bundler sign \
|
||||
--manifest manifest.json \
|
||||
--key /path/to/signing-key.pem \
|
||||
--output bundle.dsse.json
|
||||
|
||||
# Verify bundle
|
||||
stellaops-bundler verify risk-bundle-2025-12-17.tar.gz
|
||||
```
|
||||
|
||||
### Custom Bundle Scripts
|
||||
|
||||
Example for creating weekly bundles (7-day snapshots):
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# create-weekly-bundle.sh
|
||||
|
||||
WEEK_START=$(date -u -d "last monday" +%Y-%m-%d)
|
||||
WEEK_END=$(date -u +%Y-%m-%d)
|
||||
BUNDLE_ID="risk-bundle-weekly-${WEEK_START}"
|
||||
|
||||
echo "Creating weekly bundle: ${BUNDLE_ID}"
|
||||
|
||||
for day in $(seq 0 6); do
|
||||
CURRENT_DATE=$(date -u -d "${WEEK_START} + ${day} days" +%Y-%m-%d)
|
||||
# Fetch EPSS for each day...
|
||||
curl -sL "https://epss.empiricalsecurity.com/epss_scores-${CURRENT_DATE}.csv.gz" \
|
||||
-o "epss/epss_scores-${CURRENT_DATE}.csv.gz"
|
||||
done
|
||||
|
||||
# Compress and bundle...
|
||||
tar -czf "${BUNDLE_ID}.tar.gz" epss/ kev/ manifest.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-17
|
||||
**Version**: 1.0
|
||||
**Maintainer**: StellaOps Operations Team
|
||||
Reference in New Issue
Block a user