Refactor code structure for improved readability and maintainability; optimize performance in key functions.

This commit is contained in:
master
2025-12-22 19:06:31 +02:00
parent dfaa2079aa
commit 4602ccc3a3
1444 changed files with 109919 additions and 8058 deletions

View File

@@ -0,0 +1,53 @@
# Concelier Alpine SecDB Connector - Operations Runbook
_Last updated: 2025-12-22_
## 1. Overview
The Alpine connector pulls JSON secdb feeds (main/community) for configured
releases and maps CVE identifiers to APK version ranges. It preserves native
APK versions and emits `rangeKind: apk` so downstream consumers keep distro
semantics intact.
## 2. Configuration knobs (`concelier.yaml`)
```yaml
concelier:
sources:
alpine:
baseUri: "https://secdb.alpinelinux.org/"
releases:
- "v3.18"
- "v3.19"
- "v3.20"
repositories:
- "main"
- "community"
maxDocumentsPerFetch: 20
fetchTimeout: "00:00:45"
requestDelay: "00:00:00"
userAgent: "StellaOps.Concelier.Alpine/0.1 (+https://stella-ops.org)"
```
### Recommendations
- Keep `releases` to supported Alpine branches only; avoid stale branches in
production unless you maintain a mirror.
- Use `requestDelay` when running multiple source connectors on shared egress.
## 3. Default job schedule
| Job kind | Cron | Timeout | Lease |
|----------|------|---------|-------|
| `source:alpine:fetch` | `*/30 * * * *` | 5 minutes | 4 minutes |
| `source:alpine:parse` | `7,37 * * * *` | 6 minutes | 4 minutes |
| `source:alpine:map` | `12,42 * * * *` | 8 minutes | 4 minutes |
The cadence staggers fetch, parse, and map so each stage has a clean window to
complete. Override via `concelier.jobs.definitions[...]` when coordinating
multiple sources on the same scheduler.
## 4. Offline and air-gapped deployments
- Mirror `secdb` JSON files into a local repository and point `baseUri` to the
mirror host.
- The connector allowlists only the `baseUri` host; update it to match the
internal mirror host.
- Keep fixtures and exported bundles deterministic by leaving the order of
releases and repositories stable.

View File

@@ -0,0 +1,49 @@
# Concelier EPSS Connector Operations
This playbook covers deployment and monitoring of the EPSS connector that ingests daily FIRST.org EPSS snapshots.
## 1. Prerequisites
- Network egress to `https://epss.empiricalsecurity.com/` (or a mirrored endpoint).
- Updated `concelier.yaml` (or environment variables) with the EPSS source configuration:
```yaml
concelier:
sources:
epss:
baseUri: "https://epss.empiricalsecurity.com/"
fetchCurrent: true
catchUpDays: 7
httpTimeout: "00:02:00"
maxRetries: 3
airgapMode: false
bundlePath: "/var/stellaops/bundles/epss"
```
## 2. Smoke Test (staging)
1. Restart Concelier workers after configuration changes.
2. Trigger a full cycle:
- CLI: `stella db jobs run source:epss:fetch --and-then source:epss:parse --and-then source:epss:map`
- REST: `POST /jobs/run { "kind": "source:epss:fetch", "chain": ["source:epss:parse", "source:epss:map"] }`
3. Verify document status transitions: `pending_parse` -> `pending_map` -> `mapped`.
4. Confirm log entries for `Fetched EPSS snapshot` and parse/map summaries.
## 3. Monitoring
- **Meter**: `StellaOps.Concelier.Connector.Epss`
- **Key counters**:
- `epss.fetch.attempts`, `epss.fetch.success`, `epss.fetch.failures`, `epss.fetch.unchanged`
- `epss.parse.rows`, `epss.parse.failures`
- `epss.map.rows`
- **Alert suggestions**:
- `rate(epss_fetch_failures_total[15m]) > 0`
- `rate(epss_map_rows_total[1h]) == 0` during business hours while other connectors are active
## 4. Airgap Mode
- Place snapshots in the bundle directory:
- `epss_scores-YYYY-MM-DD.csv.gz`
- Optional `manifest.json` listing `name`, `modelVersion`, `sha256`, and `rowCount`.
- Set `airgapMode: true` and `bundlePath` to the directory or specific file.
- The connector validates the manifest hash when present and logs warnings on mismatch.