Concelier CERT-Bund Connector Operations

Last updated: 2025-10-17

Germany’s Federal Office for Information Security (BSI) operates the Warn- und Informationsdienst (WID) portal. The Concelier CERT-Bund connector (source:cert-bund:*) ingests the public RSS feed, hydrates the portal’s JSON detail endpoint, and maps the result into canonical advisories while preserving the original German content.

1. Configuration Checklist

Allow outbound access (or stage mirrors) for:
- https://wid.cert-bund.de/content/public/securityAdvisory/rss
- https://wid.cert-bund.de/portal/ (session/bootstrap)
- https://wid.cert-bund.de/portal/api/securityadvisory (detail/search/export JSON)
Ensure the HTTP client reuses a cookie container (the connector’s dependency injection wiring already sets this up).

Example concelier.yaml fragment:

concelier:
  sources:
    cert-bund:
      feedUri: "https://wid.cert-bund.de/content/public/securityAdvisory/rss"
      portalBootstrapUri: "https://wid.cert-bund.de/portal/"
      detailApiUri: "https://wid.cert-bund.de/portal/api/securityadvisory"
      maxAdvisoriesPerFetch: 50
      maxKnownAdvisories: 512
      requestTimeout: "00:00:30"
      requestDelay: "00:00:00.250"
      failureBackoff: "00:05:00"

Leave maxAdvisoriesPerFetch at 50 during normal operation. Raise it only for controlled backfills, then restore the default to avoid overwhelming the portal.

2. Telemetry & Logging

Meter: StellaOps.Concelier.Connector.CertBund
Counters / histograms:
- certbund.feed.fetch.attempts|success|failures
- certbund.feed.items.count
- certbund.feed.enqueued.count
- certbund.feed.coverage.days
- certbund.detail.fetch.attempts|success|not_modified|failures{reason}
- certbund.parse.success|failures{reason}
- certbund.parse.products.count, certbund.parse.cve.count
- certbund.map.success|failures{reason}
- certbund.map.affected.count, certbund.map.aliases.count
Shared HTTP metrics remain available through concelier.source.http.*.

Structured logs (all emitted at information level when work occurs):

CERT-Bund fetch cycle: … truncated {Truncated}, coverageDays={CoverageDays}
CERT-Bund parse cycle: parsed {Parsed}, failures {Failures}, …
CERT-Bund map cycle: mapped {Mapped}, failures {Failures}, …

Alerting ideas:

increase(certbund.detail.fetch.failures_total[10m]) > 0
rate(certbund.map.success_total[30m]) == 0
histogram_quantile(0.95, rate(concelier_source_http_duration_bucket{concelier_source="cert-bund"}[15m])) > 5s

The WebService now registers the meter so metrics surface automatically once OpenTelemetry metrics are enabled.

3. Historical Backfill & Export Strategy

3.1 Retention snapshot

RSS window: ~250 advisories (≈90 days at current cadence).
Older advisories are accessible through the JSON search/export APIs once the anti-CSRF token is supplied.

3.2 JSON search pagination

# 1. Bootstrap cookies (client_config + XSRF-TOKEN)
curl -s -c cookies.txt "https://wid.cert-bund.de/portal/" > /dev/null
curl -s -b cookies.txt -c cookies.txt \
     -H "X-Requested-With: XMLHttpRequest" \
     "https://wid.cert-bund.de/portal/api/security/csrf" > /dev/null

XSRF=$(awk '/XSRF-TOKEN/ {print $7}' cookies.txt)

# 2. Page search results
curl -s -b cookies.txt \
     -H "Content-Type: application/json" \
     -H "Accept: application/json" \
     -H "X-XSRF-TOKEN: ${XSRF}" \
     -X POST \
     --data '{"page":4,"size":100,"sort":["published,desc"]}' \
     "https://wid.cert-bund.de/portal/api/securityadvisory/search" \
     > certbund-page4.json

Iterate page until the response content array is empty. Pages 0–9 currently cover 2014→present. Persist JSON responses (plus SHA256) for Offline Kit parity.

Shortcut – run python tools/certbund_offline_snapshot.py --output seed-data/cert-bund to bootstrap the session, capture the paginated search responses, and regenerate the manifest/checksum files automatically. Supply --cookie-file and --xsrf-token if the portal requires a browser-derived session (see options via --help).

3.3 Export bundles

python tools/certbund_offline_snapshot.py \
  --output seed-data/cert-bund \
  --start-year 2014 \
  --end-year "$(date -u +%Y)"

The helper stores yearly exports under seed-data/cert-bund/export/, captures paginated search snapshots in seed-data/cert-bund/search/, and generates the manifest + SHA files in seed-data/cert-bund/manifest/. Split ranges according to your compliance window (default: one file per calendar year). Concelier can ingest these JSON payloads directly when operating offline.

When automatic bootstrap fails (e.g. portal introduces CAPTCHA), run the manual curl flow above, then rerun the helper with --skip-fetch to rebuild the manifest from the existing files.

3.4 Connector-driven catch-up

Temporarily raise maxAdvisoriesPerFetch (e.g. 150) and reduce requestDelay.
Run stella db jobs run source:cert-bund:fetch --and-then source:cert-bund:parse --and-then source:cert-bund:map until the fetch log reports enqueued=0.
Restore defaults and capture the cursor snapshot for audit.

4. Locale & Translation Guidance

Advisories remain in German (language: "de"). Preserve wording for provenance and legal accuracy.
UI localisation: enable the translation bundles documented in docs/15_UI_GUIDE.md if English UI copy is required. Operators can overlay machine or human translations, but the canonical database stores the source text.
Docs guild is compiling a CERT-Bund terminology glossary under docs/locale/certbund-glossary.md so downstream teams can reference consistent English equivalents without altering the stored advisories.

5. Verification Checklist

Observe certbund.feed.fetch.success and certbund.detail.fetch.success increments after runs; certbund.feed.coverage.days should hover near the observed RSS window.
Ensure summary logs report truncated=false in steady state—true indicates the fetch cap was hit.
During backfills, watch certbund.feed.enqueued.count trend to zero.
Spot-check stored advisories in Mongo to confirm language="de" and reference URLs match the portal detail endpoint.
For Offline Kit exports, validate SHA256 hashes before distribution.

6.3 KiB Raw Blame History Unescape Escape