Refactor code structure for improved readability and maintainability; optimize performance in key functions.

This commit is contained in:
master
2025-12-22 19:06:31 +02:00
parent dfaa2079aa
commit 0536a4f7d4
1443 changed files with 109671 additions and 7840 deletions

View File

@@ -0,0 +1,49 @@
# Concelier EPSS Connector Operations
This playbook covers deployment and monitoring of the EPSS connector that ingests daily FIRST.org EPSS snapshots.
## 1. Prerequisites
- Network egress to `https://epss.empiricalsecurity.com/` (or a mirrored endpoint).
- Updated `concelier.yaml` (or environment variables) with the EPSS source configuration:
```yaml
concelier:
sources:
epss:
baseUri: "https://epss.empiricalsecurity.com/"
fetchCurrent: true
catchUpDays: 7
httpTimeout: "00:02:00"
maxRetries: 3
airgapMode: false
bundlePath: "/var/stellaops/bundles/epss"
```
## 2. Smoke Test (staging)
1. Restart Concelier workers after configuration changes.
2. Trigger a full cycle:
- CLI: `stella db jobs run source:epss:fetch --and-then source:epss:parse --and-then source:epss:map`
- REST: `POST /jobs/run { "kind": "source:epss:fetch", "chain": ["source:epss:parse", "source:epss:map"] }`
3. Verify document status transitions: `pending_parse` -> `pending_map` -> `mapped`.
4. Confirm log entries for `Fetched EPSS snapshot` and parse/map summaries.
## 3. Monitoring
- **Meter**: `StellaOps.Concelier.Connector.Epss`
- **Key counters**:
- `epss.fetch.attempts`, `epss.fetch.success`, `epss.fetch.failures`, `epss.fetch.unchanged`
- `epss.parse.rows`, `epss.parse.failures`
- `epss.map.rows`
- **Alert suggestions**:
- `rate(epss_fetch_failures_total[15m]) > 0`
- `rate(epss_map_rows_total[1h]) == 0` during business hours while other connectors are active
## 4. Airgap Mode
- Place snapshots in the bundle directory:
- `epss_scores-YYYY-MM-DD.csv.gz`
- Optional `manifest.json` listing `name`, `modelVersion`, `sha256`, and `rowCount`.
- Set `airgapMode: true` and `bundlePath` to the directory or specific file.
- The connector validates the manifest hash when present and logs warnings on mismatch.