Add authority bootstrap flows and Concelier ops runbooks

2025-10-15 10:03:56 +03:00
parent ea8226120c
commit 0ba025022f
276 changed files with 21674 additions and 934 deletions
--- a/docs/ops/feedser-icscisa-operations.md
+++ b/docs/ops/feedser-icscisa-operations.md
@@ -0,0 +1,122 @@
+# Feedser CISA ICS Connector Operations
+
+This runbook documents how to provision, rotate, and validate credentials for the CISA Industrial Control Systems (ICS) connector (`source:ics-cisa:*`). Follow it before enabling the connector in staging or offline installations.
+
+## 1. Credential Provisioning
+
+1. **Create a service mailbox** reachable by the Ops crew (shared mailbox recommended).  
+2. Browse to `https://public.govdelivery.com/accounts/USDHSCISA/subscriber/new` and subscribe the mailbox to the following GovDelivery topics:
+   - `USDHSCISA_16` — ICS-CERT advisories (legacy numbering: `ICSA-YY-###`).
+   - `USDHSCISA_19` — ICS medical advisories (`ICSMA-YY-###`).
+   - `USDHSCISA_17` — ICS alerts (`IR-ALERT-YY-###`) for completeness.
+3. Complete the verification email. After confirmation, note the **personalised subscription code** included in the “Manage Preferences” link. It has the shape `code=AB12CD34EF`.
+4. Store the code in the shared secret vault (or Offline Kit secrets bundle) as `feedser/sources/icscisa/govdelivery/code`.
+
+> ℹ️  GovDelivery does not expose a one-time API key; the personalised code is what authenticates the RSS pull. Never commit it to git.
+
+## 2. Feed Validation
+
+Use the following command to confirm the feed is reachable before wiring it into Feedser (substitute `<CODE>` with the personalised value):
+
+```bash
+curl -H "User-Agent: StellaOpsFeedser/ics-cisa" \
+     "https://content.govdelivery.com/accounts/USDHSCISA/topics/ICS-CERT/feed.rss?format=xml&code=<CODE>"
+```
+
+If the endpoint returns HTTP 200 and an RSS payload, record the sample response under `docs/artifacts/icscisa/` (see Task `FEEDCONN-ICSCISA-02-007`). HTTP 403 or 406 usually means the subscription was not confirmed or the code was mistyped.
+
+## 3. Configuration Snippet
+
+Add the connector configuration to `feedser.yaml` (or equivalent environment variables):
+
+```yaml
+feedser:
+  sources:
+    icscisa:
+      govDelivery:
+        code: "${FEEDSER_ICS_CISA_GOVDELIVERY_CODE}"
+        topics:
+          - "USDHSCISA_16"
+          - "USDHSCISA_19"
+          - "USDHSCISA_17"
+      rssBaseUri: "https://content.govdelivery.com/accounts/USDHSCISA"
+      requestDelay: "00:00:01"
+      failureBackoff: "00:05:00"
+```
+
+Environment variable example:
+
+```bash
+export FEEDSER_SOURCES_ICSCISA_GOVDELIVERY_CODE="AB12CD34EF"
+```
+
+Feedser automatically register the host with the Source.Common HTTP allow-list when the connector assembly is loaded.
+
+
+Optional tuning keys (set only when needed):
+
+- `proxyUri` — HTTP/HTTPS proxy URL used when Akamai blocks direct pulls.
+- `requestVersion` / `requestVersionPolicy` — override HTTP negotiation when the proxy requires HTTP/1.1.
+- `enableDetailScrape` — toggle HTML detail fallback (defaults to true).
+- `captureAttachments` — collect PDF attachments from detail pages (defaults to true).
+- `detailBaseUri` — alternate host for detail enrichment if CISA changes their layout.
+
+## 4. Seeding Without GovDelivery
+
+If credentials are still pending, populate the connector with the community CSV dataset before enabling the live fetch:
+
+1. Run `./scripts/fetch-ics-cisa-seed.sh` (or `.ps1`) to download the latest `CISA_ICS_ADV_*.csv` files into `seed-data/ics-cisa/`.
+2. Copy the CSVs (and the generated `.sha256` files) into your Offline Kit staging area so they ship alongside the other feeds.
+3. Import the kit as usual. The connector can parse the seed data for historical context, but **live GovDelivery credentials are still required** for fresh advisories.
+4. Once credentials arrive, update `feedser:sources:icscisa:govDelivery:code` and re-trigger `source:ics-cisa:fetch` so the connector switches to the authorised feed.
+
+> The CSVs are licensed under ODbL 1.0 by the ICS Advisory Project. Preserve the attribution when redistributing them.
+
+## 4. Integration Validation
+
+1. Ensure secrets are in place and restart the Feedser workers.
+2. Run a dry-run fetch/parse/map chain against an Akamai-protected topic:
+   ```bash
+   FEEDSER_SOURCES_ICSCISA_GOVDELIVERY_CODE=... \ 
+   FEEDSER_SOURCES_ICSCISA_ENABLEDETAILSCRAPE=1 \ 
+   stella db jobs run source:ics-cisa:fetch --and-then source:ics-cisa:parse --and-then source:ics-cisa:map
+   ```
+3. Confirm logs contain `ics-cisa detail fetch` entries and that new documents/DTOs include attachments (see `docs/artifacts/icscisa`). Canonical advisories should expose PDF links as `references.kind == "attachment"` and affected packages should surface `primitives.semVer.exactValue` for single-version hits.
+4. If Akamai blocks direct fetches, set `feedser:sources:icscisa:proxyUri` to your allow-listed egress proxy and rerun the dry-run.
+
+## 4. Rotation & Incident Response
+
+- Review GovDelivery access quarterly. Rotate the personalised code whenever Ops changes the service mailbox password or membership.  
+- Revoking the subscription in GovDelivery invalidates the code immediately; update the vault and configuration in the same change.  
+- If the code leaks, remove the subscription (`https://public.govdelivery.com/accounts/USDHSCISA/subscriber/manage_preferences?code=<CODE>`), resubscribe, and distribute the new value via the vault.
+
+## 5. Offline Kit Handling
+
+Include the personalised code in `offline-kit/secrets/feedser/icscisa.env`:
+
+```
+FEEDSER_SOURCES_ICSCISA_GOVDELIVERY_CODE=AB12CD34EF
+```
+
+The Offline Kit deployment script copies this file into the container secret directory mounted at `/run/secrets/feedser`. Ensure permissions are `600` and ownership matches the Feedser runtime user.
+
+## 6. Telemetry & Monitoring
+
+The connector emits metrics under the meter `StellaOps.Feedser.Source.Ics.Cisa`. They allow operators to track Akamai fallbacks, detail enrichment health, and advisory fan-out.
+
+- `icscisa.fetch.*` – counters for `attempts`, `success`, `failures`, `not_modified`, and `fallbacks`, plus histogram `icscisa.fetch.documents` showing documents added per topic pull (tags: `feedser.source`, `icscisa.topic`).
+- `icscisa.parse.*` – counters for `success`/`failures` and histograms `icscisa.parse.advisories`, `icscisa.parse.attachments`, `icscisa.parse.detail_fetches` to monitor enrichment workload per feed document.
+- `icscisa.detail.*` – counters `success` / `failures` per advisory (tagged with `icscisa.advisory`) to alert when Akamai blocks detail pages.
+- `icscisa.map.*` – counters for `success`/`failures` and histograms `icscisa.map.references`, `icscisa.map.packages`, `icscisa.map.aliases` capturing canonical fan-out.
+
+Suggested alerts:
+
+- `increase(icscisa.fetch.failures_total[15m]) > 0` or `increase(icscisa.fetch.fallbacks_total[15m]) > 5` — sustained Akamai or proxy issues.
+- `increase(icscisa.detail.failures_total[30m]) > 0` — detail enrichment breaking (potential HTML layout change).
+- `histogram_quantile(0.95, rate(icscisa.map.references_bucket[1h]))` trending sharply higher — sudden advisory reference explosion worth investigating.
+- Keep an eye on shared HTTP metrics (`feedser.source.http.*{feedser.source="ics-cisa"}`) for request latency and retry patterns.
+
+## 6. Related Tasks
+
+- `FEEDCONN-ICSCISA-02-009` (GovDelivery credential onboarding) — completed once this runbook is followed and secrets are placed in the vault.
+- `FEEDCONN-ICSCISA-02-007` (document inventory) — archive the first successful RSS response and any attachment URL schema under `docs/artifacts/icscisa/`.