147 lines
6.3 KiB
Markdown
147 lines
6.3 KiB
Markdown
# Concelier CERT-Bund Connector Operations
|
||
|
||
_Last updated: 2025-10-17_
|
||
|
||
Germany’s Federal Office for Information Security (BSI) operates the Warn- und Informationsdienst (WID) portal. The Concelier CERT-Bund connector (`source:cert-bund:*`) ingests the public RSS feed, hydrates the portal’s JSON detail endpoint, and maps the result into canonical advisories while preserving the original German content.
|
||
|
||
---
|
||
|
||
## 1. Configuration Checklist
|
||
|
||
- Allow outbound access (or stage mirrors) for:
|
||
- `https://wid.cert-bund.de/content/public/securityAdvisory/rss`
|
||
- `https://wid.cert-bund.de/portal/` (session/bootstrap)
|
||
- `https://wid.cert-bund.de/portal/api/securityadvisory` (detail/search/export JSON)
|
||
- Ensure the HTTP client reuses a cookie container (the connector’s dependency injection wiring already sets this up).
|
||
|
||
Example `concelier.yaml` fragment:
|
||
|
||
```yaml
|
||
concelier:
|
||
sources:
|
||
cert-bund:
|
||
feedUri: "https://wid.cert-bund.de/content/public/securityAdvisory/rss"
|
||
portalBootstrapUri: "https://wid.cert-bund.de/portal/"
|
||
detailApiUri: "https://wid.cert-bund.de/portal/api/securityadvisory"
|
||
maxAdvisoriesPerFetch: 50
|
||
maxKnownAdvisories: 512
|
||
requestTimeout: "00:00:30"
|
||
requestDelay: "00:00:00.250"
|
||
failureBackoff: "00:05:00"
|
||
```
|
||
|
||
> Leave `maxAdvisoriesPerFetch` at 50 during normal operation. Raise it only for controlled backfills, then restore the default to avoid overwhelming the portal.
|
||
|
||
---
|
||
|
||
## 2. Telemetry & Logging
|
||
|
||
- **Meter**: `StellaOps.Concelier.Connector.CertBund`
|
||
- **Counters / histograms**:
|
||
- `certbund.feed.fetch.attempts|success|failures`
|
||
- `certbund.feed.items.count`
|
||
- `certbund.feed.enqueued.count`
|
||
- `certbund.feed.coverage.days`
|
||
- `certbund.detail.fetch.attempts|success|not_modified|failures{reason}`
|
||
- `certbund.parse.success|failures{reason}`
|
||
- `certbund.parse.products.count`, `certbund.parse.cve.count`
|
||
- `certbund.map.success|failures{reason}`
|
||
- `certbund.map.affected.count`, `certbund.map.aliases.count`
|
||
- Shared HTTP metrics remain available through `concelier.source.http.*`.
|
||
|
||
**Structured logs** (all emitted at information level when work occurs):
|
||
|
||
- `CERT-Bund fetch cycle: … truncated {Truncated}, coverageDays={CoverageDays}`
|
||
- `CERT-Bund parse cycle: parsed {Parsed}, failures {Failures}, …`
|
||
- `CERT-Bund map cycle: mapped {Mapped}, failures {Failures}, …`
|
||
|
||
Alerting ideas:
|
||
|
||
1. `increase(certbund.detail.fetch.failures_total[10m]) > 0`
|
||
2. `rate(certbund.map.success_total[30m]) == 0`
|
||
3. `histogram_quantile(0.95, rate(concelier_source_http_duration_bucket{concelier_source="cert-bund"}[15m])) > 5s`
|
||
|
||
The WebService now registers the meter so metrics surface automatically once OpenTelemetry metrics are enabled.
|
||
|
||
---
|
||
|
||
## 3. Historical Backfill & Export Strategy
|
||
|
||
### 3.1 Retention snapshot
|
||
|
||
- RSS window: ~250 advisories (≈90 days at current cadence).
|
||
- Older advisories are accessible through the JSON search/export APIs once the anti-CSRF token is supplied.
|
||
|
||
### 3.2 JSON search pagination
|
||
|
||
```bash
|
||
# 1. Bootstrap cookies (client_config + XSRF-TOKEN)
|
||
curl -s -c cookies.txt "https://wid.cert-bund.de/portal/" > /dev/null
|
||
curl -s -b cookies.txt -c cookies.txt \
|
||
-H "X-Requested-With: XMLHttpRequest" \
|
||
"https://wid.cert-bund.de/portal/api/security/csrf" > /dev/null
|
||
|
||
XSRF=$(awk '/XSRF-TOKEN/ {print $7}' cookies.txt)
|
||
|
||
# 2. Page search results
|
||
curl -s -b cookies.txt \
|
||
-H "Content-Type: application/json" \
|
||
-H "Accept: application/json" \
|
||
-H "X-XSRF-TOKEN: ${XSRF}" \
|
||
-X POST \
|
||
--data '{"page":4,"size":100,"sort":["published,desc"]}' \
|
||
"https://wid.cert-bund.de/portal/api/securityadvisory/search" \
|
||
> certbund-page4.json
|
||
```
|
||
|
||
Iterate `page` until the response `content` array is empty. Pages 0–9 currently cover 2014→present. Persist JSON responses (plus SHA256) for Offline Kit parity.
|
||
|
||
> **Shortcut** – run `python tools/certbund_offline_snapshot.py --output seed-data/cert-bund`
|
||
> to bootstrap the session, capture the paginated search responses, and regenerate
|
||
> the manifest/checksum files automatically. Supply `--cookie-file` and `--xsrf-token`
|
||
> if the portal requires a browser-derived session (see options via `--help`).
|
||
|
||
### 3.3 Export bundles
|
||
|
||
```bash
|
||
python tools/certbund_offline_snapshot.py \
|
||
--output seed-data/cert-bund \
|
||
--start-year 2014 \
|
||
--end-year "$(date -u +%Y)"
|
||
```
|
||
|
||
The helper stores yearly exports under `seed-data/cert-bund/export/`,
|
||
captures paginated search snapshots in `seed-data/cert-bund/search/`,
|
||
and generates the manifest + SHA files in `seed-data/cert-bund/manifest/`.
|
||
Split ranges according to your compliance window (default: one file per
|
||
calendar year). Concelier can ingest these JSON payloads directly when
|
||
operating offline.
|
||
|
||
> When automatic bootstrap fails (e.g. portal introduces CAPTCHA), run the
|
||
> manual `curl` flow above, then rerun the helper with `--skip-fetch` to
|
||
> rebuild the manifest from the existing files.
|
||
|
||
### 3.4 Connector-driven catch-up
|
||
|
||
1. Temporarily raise `maxAdvisoriesPerFetch` (e.g. 150) and reduce `requestDelay`.
|
||
2. Run `stella db jobs run source:cert-bund:fetch --and-then source:cert-bund:parse --and-then source:cert-bund:map` until the fetch log reports `enqueued=0`.
|
||
3. Restore defaults and capture the cursor snapshot for audit.
|
||
|
||
---
|
||
|
||
## 4. Locale & Translation Guidance
|
||
|
||
- Advisories remain in German (`language: "de"`). Preserve wording for provenance and legal accuracy.
|
||
- UI localisation: enable the translation bundles documented in `docs/15_UI_GUIDE.md` if English UI copy is required. Operators can overlay machine or human translations, but the canonical database stores the source text.
|
||
- Docs guild is compiling a CERT-Bund terminology glossary under `docs/locale/certbund-glossary.md` so downstream teams can reference consistent English equivalents without altering the stored advisories.
|
||
|
||
---
|
||
|
||
## 5. Verification Checklist
|
||
|
||
1. Observe `certbund.feed.fetch.success` and `certbund.detail.fetch.success` increments after runs; `certbund.feed.coverage.days` should hover near the observed RSS window.
|
||
2. Ensure summary logs report `truncated=false` in steady state—`true` indicates the fetch cap was hit.
|
||
3. During backfills, watch `certbund.feed.enqueued.count` trend to zero.
|
||
4. Spot-check stored advisories in Mongo to confirm `language="de"` and reference URLs match the portal detail endpoint.
|
||
5. For Offline Kit exports, validate SHA256 hashes before distribution.
|