Refactor code structure for improved readability and maintainability; optimize performance in key functions.
This commit is contained in:
7
docs/modules/concelier/connectors.md
Normal file
7
docs/modules/concelier/connectors.md
Normal file
@@ -0,0 +1,7 @@
|
||||
# Concelier Connectors
|
||||
|
||||
This index lists Concelier connectors and links to their operational runbooks. For detailed procedures and alerting, see `docs/modules/concelier/operations/connectors/`.
|
||||
|
||||
| Connector | Source ID | Purpose | Ops Runbook |
|
||||
| --- | --- | --- | --- |
|
||||
| EPSS | `epss` | FIRST.org EPSS exploitation probability feed | `docs/modules/concelier/operations/connectors/epss.md` |
|
||||
53
docs/modules/concelier/operations/connectors/alpine.md
Normal file
53
docs/modules/concelier/operations/connectors/alpine.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# Concelier Alpine SecDB Connector - Operations Runbook
|
||||
|
||||
_Last updated: 2025-12-22_
|
||||
|
||||
## 1. Overview
|
||||
The Alpine connector pulls JSON secdb feeds (main/community) for configured
|
||||
releases and maps CVE identifiers to APK version ranges. It preserves native
|
||||
APK versions and emits `rangeKind: apk` so downstream consumers keep distro
|
||||
semantics intact.
|
||||
|
||||
## 2. Configuration knobs (`concelier.yaml`)
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
alpine:
|
||||
baseUri: "https://secdb.alpinelinux.org/"
|
||||
releases:
|
||||
- "v3.18"
|
||||
- "v3.19"
|
||||
- "v3.20"
|
||||
repositories:
|
||||
- "main"
|
||||
- "community"
|
||||
maxDocumentsPerFetch: 20
|
||||
fetchTimeout: "00:00:45"
|
||||
requestDelay: "00:00:00"
|
||||
userAgent: "StellaOps.Concelier.Alpine/0.1 (+https://stella-ops.org)"
|
||||
```
|
||||
|
||||
### Recommendations
|
||||
- Keep `releases` to supported Alpine branches only; avoid stale branches in
|
||||
production unless you maintain a mirror.
|
||||
- Use `requestDelay` when running multiple source connectors on shared egress.
|
||||
|
||||
## 3. Default job schedule
|
||||
|
||||
| Job kind | Cron | Timeout | Lease |
|
||||
|----------|------|---------|-------|
|
||||
| `source:alpine:fetch` | `*/30 * * * *` | 5 minutes | 4 minutes |
|
||||
| `source:alpine:parse` | `7,37 * * * *` | 6 minutes | 4 minutes |
|
||||
| `source:alpine:map` | `12,42 * * * *` | 8 minutes | 4 minutes |
|
||||
|
||||
The cadence staggers fetch, parse, and map so each stage has a clean window to
|
||||
complete. Override via `concelier.jobs.definitions[...]` when coordinating
|
||||
multiple sources on the same scheduler.
|
||||
|
||||
## 4. Offline and air-gapped deployments
|
||||
- Mirror `secdb` JSON files into a local repository and point `baseUri` to the
|
||||
mirror host.
|
||||
- The connector allowlists only the `baseUri` host; update it to match the
|
||||
internal mirror host.
|
||||
- Keep fixtures and exported bundles deterministic by leaving the order of
|
||||
releases and repositories stable.
|
||||
49
docs/modules/concelier/operations/connectors/epss.md
Normal file
49
docs/modules/concelier/operations/connectors/epss.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# Concelier EPSS Connector Operations
|
||||
|
||||
This playbook covers deployment and monitoring of the EPSS connector that ingests daily FIRST.org EPSS snapshots.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Network egress to `https://epss.empiricalsecurity.com/` (or a mirrored endpoint).
|
||||
- Updated `concelier.yaml` (or environment variables) with the EPSS source configuration:
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
epss:
|
||||
baseUri: "https://epss.empiricalsecurity.com/"
|
||||
fetchCurrent: true
|
||||
catchUpDays: 7
|
||||
httpTimeout: "00:02:00"
|
||||
maxRetries: 3
|
||||
airgapMode: false
|
||||
bundlePath: "/var/stellaops/bundles/epss"
|
||||
```
|
||||
|
||||
## 2. Smoke Test (staging)
|
||||
|
||||
1. Restart Concelier workers after configuration changes.
|
||||
2. Trigger a full cycle:
|
||||
- CLI: `stella db jobs run source:epss:fetch --and-then source:epss:parse --and-then source:epss:map`
|
||||
- REST: `POST /jobs/run { "kind": "source:epss:fetch", "chain": ["source:epss:parse", "source:epss:map"] }`
|
||||
3. Verify document status transitions: `pending_parse` -> `pending_map` -> `mapped`.
|
||||
4. Confirm log entries for `Fetched EPSS snapshot` and parse/map summaries.
|
||||
|
||||
## 3. Monitoring
|
||||
|
||||
- **Meter**: `StellaOps.Concelier.Connector.Epss`
|
||||
- **Key counters**:
|
||||
- `epss.fetch.attempts`, `epss.fetch.success`, `epss.fetch.failures`, `epss.fetch.unchanged`
|
||||
- `epss.parse.rows`, `epss.parse.failures`
|
||||
- `epss.map.rows`
|
||||
- **Alert suggestions**:
|
||||
- `rate(epss_fetch_failures_total[15m]) > 0`
|
||||
- `rate(epss_map_rows_total[1h]) == 0` during business hours while other connectors are active
|
||||
|
||||
## 4. Airgap Mode
|
||||
|
||||
- Place snapshots in the bundle directory:
|
||||
- `epss_scores-YYYY-MM-DD.csv.gz`
|
||||
- Optional `manifest.json` listing `name`, `modelVersion`, `sha256`, and `rowCount`.
|
||||
- Set `airgapMode: true` and `bundlePath` to the directory or specific file.
|
||||
- The connector validates the manifest hash when present and logs warnings on mismatch.
|
||||
Reference in New Issue
Block a user