Add authority bootstrap flows and Concelier ops runbooks

2025-10-15 10:03:56 +03:00
parent 0ddc014864
commit bab75fb00d
276 changed files with 21674 additions and 934 deletions
--- a/docs/ops/feedser-cccs-operations.md
+++ b/docs/ops/feedser-cccs-operations.md
@@ -0,0 +1,72 @@
+# Feedser CCCS Connector Operations
+
+This runbook covers day‑to‑day operation of the Canadian Centre for Cyber Security (`source:cccs:*`) connector, including configuration, telemetry, and historical backfill guidance for English/French advisories.
+
+## 1. Configuration Checklist
+
+- Network egress (or mirrored cache) for `https://www.cyber.gc.ca/` and the JSON API endpoints under `/api/cccs/`.
+- Set the Feedser options before restarting workers. Example `feedser.yaml` snippet:
+
+```yaml
+feedser:
+  sources:
+    cccs:
+      feeds:
+        - language: "en"
+          uri: "https://www.cyber.gc.ca/api/cccs/threats/v1/get?lang=en&content_type=cccs_threat"
+        - language: "fr"
+          uri: "https://www.cyber.gc.ca/api/cccs/threats/v1/get?lang=fr&content_type=cccs_threat"
+      maxEntriesPerFetch: 80        # increase temporarily for backfill runs
+      maxKnownEntries: 512
+      requestTimeout: "00:00:30"
+      requestDelay: "00:00:00.250"
+      failureBackoff: "00:05:00"
+```
+
+> ℹ️  The `/api/cccs/threats/v1/get` endpoint returns thousands of records per language (≈5 100 rows each as of 2025‑10‑14). The connector honours `maxEntriesPerFetch`, so leave it low for steady‑state and raise it for planned backfills.
+
+## 2. Telemetry & Logging
+
+- **Metrics (Meter `StellaOps.Feedser.Source.Cccs`):**
+  - `cccs.fetch.attempts`, `cccs.fetch.success`, `cccs.fetch.failures`
+  - `cccs.fetch.documents`, `cccs.fetch.unchanged`
+  - `cccs.parse.success`, `cccs.parse.failures`, `cccs.parse.quarantine`
+  - `cccs.map.success`, `cccs.map.failures`
+- **Shared HTTP metrics** via `SourceDiagnostics`:
+  - `feedser.source.http.requests{feedser.source="cccs"}`
+  - `feedser.source.http.failures{feedser.source="cccs"}`
+  - `feedser.source.http.duration{feedser.source="cccs"}`
+- **Structured logs**
+  - `CCCS fetch completed feeds=… items=… newDocuments=… pendingDocuments=…`
+  - `CCCS parse completed parsed=… failures=…`
+  - `CCCS map completed mapped=… failures=…`
+  - Warnings fire when GridFS payloads/DTOs go missing or parser sanitisation fails.
+
+Suggested Grafana alerts:
+- `increase(cccs.fetch.failures_total[15m]) > 0`
+- `rate(cccs.map.success_total[1h]) == 0` while other connectors are active
+- `histogram_quantile(0.95, rate(feedser_source_http_duration_bucket{feedser_source="cccs"}[1h])) > 5s`
+
+## 3. Historical Backfill Plan
+
+1. **Snapshot the source** – the API accepts `page=<n>` and `lang=<en|fr>` query parameters. `page=0` returns the full dataset (observed earliest `date_created`: 2018‑06‑08 for EN, 2018‑06‑08 for FR). Mirror those responses into Offline Kit storage when operating air‑gapped.
+2. **Stage ingestion**:
+   - Temporarily raise `maxEntriesPerFetch` (e.g. 500) and restart Feedser workers.
+   - Run chained jobs until `pendingDocuments` drains:  
+     `stella db jobs run source:cccs:fetch --and-then source:cccs:parse --and-then source:cccs:map`
+   - Monitor `cccs.fetch.unchanged` growth; once it approaches dataset size the backfill is complete.
+3. **Optional pagination sweep** – for incremental mirrors, iterate `page=<n>` (0…N) while `response.Count == 50`, persisting JSON to disk. Store alongside metadata (`language`, `page`, SHA256) so repeated runs detect drift.
+4. **Language split** – keep EN/FR payloads separate to preserve canonical language fields. The connector emits `Language` directly from the feed entry, so mixed ingestion simply produces parallel advisories keyed by the same serial number.
+5. **Throttle planning** – schedule backfills during maintenance windows; the API tolerates burst downloads but respect the 250 ms request delay or raise it if mirrored traffic is not available.
+
+## 4. Selector & Sanitiser Notes
+
+- `CccsHtmlParser` now parses the **unsanitised DOM** (via AngleSharp) and only sanitises when persisting `ContentHtml`.
+- Product extraction walks headings (`Affected Products`, `Produits touchés`, `Mesures recommandées`) and consumes nested lists within `div/section/article` containers.
+- `HtmlContentSanitizer` allows `<h1>…<h6>` and `<section>` so stored HTML keeps headings for UI rendering and downstream summarisation.
+
+## 5. Fixture Maintenance
+
+- Regression fixtures live in `src/StellaOps.Feedser.Source.Cccs.Tests/Fixtures`.
+- Refresh via `UPDATE_CCCS_FIXTURES=1 dotnet test src/StellaOps.Feedser.Source.Cccs.Tests/StellaOps.Feedser.Source.Cccs.Tests.csproj`.
+- Fixtures capture both EN/FR advisories with nested lists to guard against sanitiser regressions; review diffs for heading/list changes before committing.
--- a/docs/ops/feedser-certbund-operations.md
+++ b/docs/ops/feedser-certbund-operations.md
@@ -0,0 +1,134 @@
+# Feedser CERT-Bund Connector Operations
+
+_Last updated: 2025-10-15_
+
+Germany’s Federal Office for Information Security (BSI) operates the Warn- und Informationsdienst (WID) portal. The Feedser CERT-Bund connector (`source:cert-bund:*`) ingests the public RSS feed, hydrates the portal’s JSON detail endpoint, and maps the result into canonical advisories while preserving the original German content.
+
+---
+
+## 1. Configuration Checklist
+
+- Allow outbound access (or stage mirrors) for:
+  - `https://wid.cert-bund.de/content/public/securityAdvisory/rss`
+  - `https://wid.cert-bund.de/portal/` (session/bootstrap)
+  - `https://wid.cert-bund.de/portal/api/securityadvisory` (detail/search/export JSON)
+- Ensure the HTTP client reuses a cookie container (the connector’s dependency injection wiring already sets this up).
+
+Example `feedser.yaml` fragment:
+
+```yaml
+feedser:
+  sources:
+    cert-bund:
+      feedUri: "https://wid.cert-bund.de/content/public/securityAdvisory/rss"
+      portalBootstrapUri: "https://wid.cert-bund.de/portal/"
+      detailApiUri: "https://wid.cert-bund.de/portal/api/securityadvisory"
+      maxAdvisoriesPerFetch: 50
+      maxKnownAdvisories: 512
+      requestTimeout: "00:00:30"
+      requestDelay: "00:00:00.250"
+      failureBackoff: "00:05:00"
+```
+
+> Leave `maxAdvisoriesPerFetch` at 50 during normal operation. Raise it only for controlled backfills, then restore the default to avoid overwhelming the portal.
+
+---
+
+## 2. Telemetry & Logging
+
+- **Meter**: `StellaOps.Feedser.Source.CertBund`
+- **Counters / histograms**:
+  - `certbund.feed.fetch.attempts|success|failures`
+  - `certbund.feed.items.count`
+  - `certbund.feed.enqueued.count`
+  - `certbund.feed.coverage.days`
+  - `certbund.detail.fetch.attempts|success|not_modified|failures{reason}`
+  - `certbund.parse.success|failures{reason}`
+  - `certbund.parse.products.count`, `certbund.parse.cve.count`
+  - `certbund.map.success|failures{reason}`
+  - `certbund.map.affected.count`, `certbund.map.aliases.count`
+- Shared HTTP metrics remain available through `feedser.source.http.*`.
+
+**Structured logs** (all emitted at information level when work occurs):
+
+- `CERT-Bund fetch cycle: … truncated {Truncated}, coverageDays={CoverageDays}`
+- `CERT-Bund parse cycle: parsed {Parsed}, failures {Failures}, …`
+- `CERT-Bund map cycle: mapped {Mapped}, failures {Failures}, …`
+
+Alerting ideas:
+
+1. `increase(certbund.detail.fetch.failures_total[10m]) > 0`
+2. `rate(certbund.map.success_total[30m]) == 0`
+3. `histogram_quantile(0.95, rate(feedser_source_http_duration_bucket{feedser_source="cert-bund"}[15m])) > 5s`
+
+The WebService now registers the meter so metrics surface automatically once OpenTelemetry metrics are enabled.
+
+---
+
+## 3. Historical Backfill & Export Strategy
+
+### 3.1 Retention snapshot
+
+- RSS window: ~250 advisories (≈90 days at current cadence).
+- Older advisories are accessible through the JSON search/export APIs once the anti-CSRF token is supplied.
+
+### 3.2 JSON search pagination
+
+```bash
+# 1. Bootstrap cookies (client_config + XSRF-TOKEN)
+curl -s -c cookies.txt "https://wid.cert-bund.de/portal/" > /dev/null
+curl -s -b cookies.txt -c cookies.txt \
+     -H "X-Requested-With: XMLHttpRequest" \
+     "https://wid.cert-bund.de/portal/api/security/csrf" > /dev/null
+
+XSRF=$(awk '/XSRF-TOKEN/ {print $7}' cookies.txt)
+
+# 2. Page search results
+curl -s -b cookies.txt \
+     -H "Content-Type: application/json" \
+     -H "Accept: application/json" \
+     -H "X-XSRF-TOKEN: ${XSRF}" \
+     -X POST \
+     --data '{"page":4,"size":100,"sort":["published,desc"]}' \
+     "https://wid.cert-bund.de/portal/api/securityadvisory/search" \
+     > certbund-page4.json
+```
+
+Iterate `page` until the response `content` array is empty. Pages 0–9 currently cover 2014→present. Persist JSON responses (plus SHA256) for Offline Kit parity.
+
+### 3.3 Export bundles
+
+```bash
+curl -s -b cookies.txt \
+     -H "Accept: application/json" \
+     -H "X-XSRF-TOKEN: ${XSRF}" \
+     "https://wid.cert-bund.de/portal/api/securityadvisory/export?format=json&from=2020-01-01" \
+     > certbund-2020-2025.json
+```
+
+Split long ranges per year and record provenance (`from`, `to`, SHA, capturedAt). Feedser can ingest these JSON payloads directly when operating offline.
+Task `FEEDCONN-CERTBUND-02-009` tracks turning this workflow into a shipped Offline Kit artefact with manifests and documentation updates—coordinate with the Docs guild before publishing.
+
+### 3.4 Connector-driven catch-up
+
+1. Temporarily raise `maxAdvisoriesPerFetch` (e.g. 150) and reduce `requestDelay`.
+2. Run `stella db jobs run source:cert-bund:fetch --and-then source:cert-bund:parse --and-then source:cert-bund:map` until the fetch log reports `enqueued=0`.
+3. Restore defaults and capture the cursor snapshot for audit.
+
+---
+
+## 4. Locale & Translation Guidance
+
+- Advisories remain in German (`language: "de"`). Preserve wording for provenance and legal accuracy.
+- UI localisation: enable the translation bundles documented in `docs/15_UI_GUIDE.md` if English UI copy is required. Operators can overlay machine or human translations, but the canonical database stores the source text.
+- Docs guild is compiling a CERT-Bund terminology glossary under `docs/locale/certbund-glossary.md` so downstream teams can reference consistent English equivalents without altering the stored advisories.
+
+---
+
+## 5. Verification Checklist
+
+1. Observe `certbund.feed.fetch.success` and `certbund.detail.fetch.success` increments after runs; `certbund.feed.coverage.days` should hover near the observed RSS window.
+2. Ensure summary logs report `truncated=false` in steady state—`true` indicates the fetch cap was hit.
+3. During backfills, watch `certbund.feed.enqueued.count` trend to zero.
+4. Spot-check stored advisories in Mongo to confirm `language="de"` and reference URLs match the portal detail endpoint.
+5. For Offline Kit exports, validate SHA256 hashes before distribution.
--- a/docs/ops/feedser-cisco-operations.md
+++ b/docs/ops/feedser-cisco-operations.md
@@ -0,0 +1,94 @@
+# Feedser Cisco PSIRT Connector – OAuth Provisioning SOP
+
+_Last updated: 2025-10-14_
+
+## 1. Scope
+
+This runbook describes how Ops provisions, rotates, and distributes Cisco PSIRT openVuln OAuth client credentials for the Feedser Cisco connector. It covers online and air-gapped (Offline Kit) environments, quota-aware execution, and escalation paths.
+
+## 2. Prerequisites
+
+- Active Cisco.com (CCO) account with access to the Cisco API Console.
+- Cisco PSIRT openVuln API entitlement (visible under “My Apps & Keys” once granted).citeturn3search0
+- Feedser configuration location (typically `/etc/stella/feedser.yaml` in production) or Offline Kit secret bundle staging directory.
+
+## 3. Provisioning workflow
+
+1. **Register the application**
+   - Sign in at <https://apiconsole.cisco.com>.
+   - Select **Register a New App** → Application Type: `Service`, Grant Type: `Client Credentials`, API: `Cisco PSIRT openVuln API`.citeturn3search0
+   - Record the generated `clientId` and `clientSecret` in the Ops vault.
+2. **Verify token issuance**
+   - Request an access token with:
+     ```bash
+     curl -s https://id.cisco.com/oauth2/default/v1/token \
+       -H "Content-Type: application/x-www-form-urlencoded" \
+       -d "grant_type=client_credentials" \
+       -d "client_id=${CLIENT_ID}" \
+       -d "client_secret=${CLIENT_SECRET}"
+     ```
+   - Confirm HTTP 200 and an `expires_in` value of 3600 seconds (tokens live for one hour).citeturn3search0turn3search7
+   - Preserve the response only long enough to validate syntax; do **not** persist tokens.
+3. **Authorize Feedser runtime**
+   - Update `feedser:sources:cisco:auth` (or the module-specific secret template) with the stored credentials.
+   - For Offline Kit delivery, export encrypted secrets into `offline-kit/secrets/cisco-openvuln.json` using the platform’s sealed secret format.
+4. **Connectivity validation**
+   - From the Feedser control plane, run `stella db jobs run source:vndr-cisco:fetch --dry-run`.
+   - Ensure the Source HTTP diagnostics record `Bearer` authorization headers and no 401/403 responses.
+
+## 4. Rotation SOP
+
+| Step | Owner | Notes |
+| --- | --- | --- |
+| 1. Schedule rotation | Ops (monthly board) | Rotate every 90 days or immediately after suspected credential exposure. |
+| 2. Create replacement app | Ops | Repeat §3.1 with “-next” suffix; verify token issuance. |
+| 3. Stage dual credentials | Ops + Feedser On-Call | Publish new credentials to secret store alongside current pair. |
+| 4. Cut over | Feedser On-Call | Restart connector workers during a low-traffic window (<10 min) to pick up the new secret. |
+| 5. Deactivate legacy app | Ops | Delete prior app in Cisco API Console once telemetry confirms successful fetch/parse cycles for 2 consecutive hours. |
+
+**Automation hooks**
+- Rotation reminders are tracked in OpsRunbookOps board (`OPS-RUN-KEYS` swim lane); add checklist items for Feedser Cisco when opening a rotation task.
+- Use the secret management pipeline (`ops/secrets/rotate.sh --connector cisco`) to template vault updates; the script renders a redacted diff for audit.
+
+## 5. Offline Kit packaging
+
+1. Generate the credential bundle using the Offline Kit CLI:  
+   `offline-kit secrets add cisco-openvuln --client-id … --client-secret …`
+2. Store the encrypted payload under `offline-kit/secrets/cisco-openvuln.enc`.
+3. Distribute via the Offline Kit channel; update `offline-kit/MANIFEST.md` with the credential fingerprint (SHA256 of plaintext concatenated with metadata).
+4. Document validation steps for the receiving site (token request from an air-gapped relay or cached token mirror).
+
+## 6. Quota and throttling guidance
+
+- Cisco enforces combined limits of 5 requests/second, 30 requests/minute, and 5 000 requests/day per application.citeturn0search0turn3search6
+- Feedser fetch jobs must respect `Retry-After` headers on HTTP 429 responses; Ops should monitor for sustained quota saturation and consider paging window adjustments.
+- Telemetry to watch: `feedser.source.http.requests{feedser.source="vndr-cisco"}`, `feedser.source.http.failures{...}`, and connector-specific metrics once implemented.
+
+## 7. Telemetry & Monitoring
+
+- **Metrics (Meter `StellaOps.Feedser.Source.Vndr.Cisco`)**
+  - `cisco.fetch.documents`, `cisco.fetch.failures`, `cisco.fetch.unchanged`
+  - `cisco.parse.success`, `cisco.parse.failures`
+  - `cisco.map.success`, `cisco.map.failures`, `cisco.map.affected.packages`
+- **Shared HTTP metrics** via `SourceDiagnostics`:
+  - `feedser.source.http.requests{feedser.source="vndr-cisco"}`
+  - `feedser.source.http.failures{feedser.source="vndr-cisco"}`
+  - `feedser.source.http.duration{feedser.source="vndr-cisco"}`
+- **Structured logs**
+  - `Cisco fetch completed date=… pages=… added=…` (info)
+  - `Cisco parse completed parsed=… failures=…` (info)
+  - `Cisco map completed mapped=… failures=…` (info)
+  - Warnings surface when DTO serialization fails or GridFS payload is missing.
+- Suggested alerts: non-zero `cisco.fetch.failures` in 15m, or `cisco.map.success` flatlines while fetch continues.
+
+## 8. Incident response
+
+- **Token compromise** – revoke the application in the Cisco API Console, purge cached secrets, rotate immediately per §4.
+- **Persistent 401/403** – confirm credentials in vault, then validate token issuance; if unresolved, open a Cisco DevNet support ticket referencing the application ID.
+- **429 spikes** – inspect job scheduler cadence and adjust connector options (`maxRequestsPerWindow`) before requesting higher quotas from Cisco.
+
+## 9. References
+
+- Cisco PSIRT openVuln API Authentication Guide.citeturn3search0
+- Accessing the openVuln API using curl (token lifetime).citeturn3search7
+- openVuln API rate limit documentation.citeturn0search0turn3search6
--- a/docs/ops/feedser-conflict-resolution.md
+++ b/docs/ops/feedser-conflict-resolution.md
@@ -150,3 +150,11 @@ dotnet test src/StellaOps.Feedser.Merge.Tests/StellaOps.Feedser.Merge.Tests.cspr
 ```

 - **Expected signals** – The triple produces one freshness-driven summary override (`primary_source=osv`, `suppressed_source=ghsa`) and one range override for the npm SemVer package while leaving `feedser.merge.conflicts` at zero. Use these values as the baseline when tuning dashboards or load-testing alert pipelines.
+
+---
+
+## 10. Change Log
+
+| Date (UTC) | Change | Notes |
+|------------|--------|-------|
+| 2025-10-16 | Ops review signed off after connector expansion (CCCS, CERT-Bund, KISA, ICS CISA, MSRC) landed. Alert thresholds from §3 reaffirmed; dashboards updated to watch attachment signals emitted by ICS CISA connector. | Ops sign-off recorded by Feedser Ops Guild; no additional overrides required. |
--- a/docs/ops/feedser-cve-kev-operations.md
+++ b/docs/ops/feedser-cve-kev-operations.md
@@ -18,6 +18,7 @@ feedser:
      apiOrg: "ORG123"
      apiUser: "user@example.org"
      apiKeyFile: "/var/run/secrets/feedser/cve-api-key"
+      seedDirectory: "./seed-data/cve"
      pageSize: 200
      maxPagesPerFetch: 5
      initialBackfill: "30.00:00:00"
@@ -27,6 +28,8 @@ feedser:

 > ℹ️  Store the API key outside source control. When using `apiKeyFile`, mount the secret file into the container/host; alternatively supply `apiKey` via `FEEDSER_SOURCES__CVE__APIKEY`.

+> 🪙  When credentials are not yet available, configure `seedDirectory` to point at mirrored CVE JSON (for example, the repo’s `seed-data/cve/` bundle). The connector will ingest those records and log a warning instead of failing the job; live fetching resumes automatically once `apiOrg` / `apiUser` / `apiKey` are supplied.
+
 ### 1.2 Smoke Test (staging)

 1. Deploy the updated configuration and restart the Feedser service so the connector picks up the credentials.
@@ -51,6 +54,26 @@ feedser:
 - **Grafana pack** – Import `docs/ops/feedser-cve-kev-grafana-dashboard.json` and filter by panel legend (`CVE`, `KEV`) to reuse the canned layout.
 - **Backfill window** – Operators can tighten or widen `initialBackfill` / `maxPagesPerFetch` after validating throughput. Update config and restart Feedser to apply changes.

+### 1.4 Staging smoke log (2025-10-15)
+
+While Ops finalises long-lived CVE Services credentials, we validated the connector end-to-end against the recorded CVE-2024-0001 payloads used in regression tests:
+
+- Command: `dotnet test src/StellaOps.Feedser.Source.Cve.Tests/StellaOps.Feedser.Source.Cve.Tests.csproj -l "console;verbosity=detailed"`
+- Summary log emitted by the connector:
+  ```
+  CVEs fetch window 2024-09-01T00:00:00Z->2024-10-01T00:00:00Z pages=1 listSuccess=1 detailDocuments=1 detailFailures=0 detailUnchanged=0 pendingDocuments=0->1 pendingMappings=0->1 hasMorePages=False nextWindowStart=2024-09-15T12:00:00Z nextWindowEnd=(none) nextPage=1
+  ```
+- Telemetry captured by `Meter` `StellaOps.Feedser.Source.Cve`:
+  | Metric | Value |
+  |--------|-------|
+  | `cve.fetch.attempts` | 1 |
+  | `cve.fetch.success` | 1 |
+  | `cve.fetch.documents` | 1 |
+  | `cve.parse.success` | 1 |
+  | `cve.map.success` | 1 |
+
+The Grafana pack `docs/ops/feedser-cve-kev-grafana-dashboard.json` has been imported into staging so the panels referenced above render against these counters once the live API keys are in place.
+
 ## 2. CISA KEV Connector (`source:kev:*`)

 ### 2.1 Prerequisites
--- a/docs/ops/feedser-icscisa-operations.md
+++ b/docs/ops/feedser-icscisa-operations.md
@@ -0,0 +1,122 @@
+# Feedser CISA ICS Connector Operations
+
+This runbook documents how to provision, rotate, and validate credentials for the CISA Industrial Control Systems (ICS) connector (`source:ics-cisa:*`). Follow it before enabling the connector in staging or offline installations.
+
+## 1. Credential Provisioning
+
+1. **Create a service mailbox** reachable by the Ops crew (shared mailbox recommended).  
+2. Browse to `https://public.govdelivery.com/accounts/USDHSCISA/subscriber/new` and subscribe the mailbox to the following GovDelivery topics:
+   - `USDHSCISA_16` — ICS-CERT advisories (legacy numbering: `ICSA-YY-###`).
+   - `USDHSCISA_19` — ICS medical advisories (`ICSMA-YY-###`).
+   - `USDHSCISA_17` — ICS alerts (`IR-ALERT-YY-###`) for completeness.
+3. Complete the verification email. After confirmation, note the **personalised subscription code** included in the “Manage Preferences” link. It has the shape `code=AB12CD34EF`.
+4. Store the code in the shared secret vault (or Offline Kit secrets bundle) as `feedser/sources/icscisa/govdelivery/code`.
+
+> ℹ️  GovDelivery does not expose a one-time API key; the personalised code is what authenticates the RSS pull. Never commit it to git.
+
+## 2. Feed Validation
+
+Use the following command to confirm the feed is reachable before wiring it into Feedser (substitute `<CODE>` with the personalised value):
+
+```bash
+curl -H "User-Agent: StellaOpsFeedser/ics-cisa" \
+     "https://content.govdelivery.com/accounts/USDHSCISA/topics/ICS-CERT/feed.rss?format=xml&code=<CODE>"
+```
+
+If the endpoint returns HTTP 200 and an RSS payload, record the sample response under `docs/artifacts/icscisa/` (see Task `FEEDCONN-ICSCISA-02-007`). HTTP 403 or 406 usually means the subscription was not confirmed or the code was mistyped.
+
+## 3. Configuration Snippet
+
+Add the connector configuration to `feedser.yaml` (or equivalent environment variables):
+
+```yaml
+feedser:
+  sources:
+    icscisa:
+      govDelivery:
+        code: "${FEEDSER_ICS_CISA_GOVDELIVERY_CODE}"
+        topics:
+          - "USDHSCISA_16"
+          - "USDHSCISA_19"
+          - "USDHSCISA_17"
+      rssBaseUri: "https://content.govdelivery.com/accounts/USDHSCISA"
+      requestDelay: "00:00:01"
+      failureBackoff: "00:05:00"
+```
+
+Environment variable example:
+
+```bash
+export FEEDSER_SOURCES_ICSCISA_GOVDELIVERY_CODE="AB12CD34EF"
+```
+
+Feedser automatically register the host with the Source.Common HTTP allow-list when the connector assembly is loaded.
+
+
+Optional tuning keys (set only when needed):
+
+- `proxyUri` — HTTP/HTTPS proxy URL used when Akamai blocks direct pulls.
+- `requestVersion` / `requestVersionPolicy` — override HTTP negotiation when the proxy requires HTTP/1.1.
+- `enableDetailScrape` — toggle HTML detail fallback (defaults to true).
+- `captureAttachments` — collect PDF attachments from detail pages (defaults to true).
+- `detailBaseUri` — alternate host for detail enrichment if CISA changes their layout.
+
+## 4. Seeding Without GovDelivery
+
+If credentials are still pending, populate the connector with the community CSV dataset before enabling the live fetch:
+
+1. Run `./scripts/fetch-ics-cisa-seed.sh` (or `.ps1`) to download the latest `CISA_ICS_ADV_*.csv` files into `seed-data/ics-cisa/`.
+2. Copy the CSVs (and the generated `.sha256` files) into your Offline Kit staging area so they ship alongside the other feeds.
+3. Import the kit as usual. The connector can parse the seed data for historical context, but **live GovDelivery credentials are still required** for fresh advisories.
+4. Once credentials arrive, update `feedser:sources:icscisa:govDelivery:code` and re-trigger `source:ics-cisa:fetch` so the connector switches to the authorised feed.
+
+> The CSVs are licensed under ODbL 1.0 by the ICS Advisory Project. Preserve the attribution when redistributing them.
+
+## 4. Integration Validation
+
+1. Ensure secrets are in place and restart the Feedser workers.
+2. Run a dry-run fetch/parse/map chain against an Akamai-protected topic:
+   ```bash
+   FEEDSER_SOURCES_ICSCISA_GOVDELIVERY_CODE=... \ 
+   FEEDSER_SOURCES_ICSCISA_ENABLEDETAILSCRAPE=1 \ 
+   stella db jobs run source:ics-cisa:fetch --and-then source:ics-cisa:parse --and-then source:ics-cisa:map
+   ```
+3. Confirm logs contain `ics-cisa detail fetch` entries and that new documents/DTOs include attachments (see `docs/artifacts/icscisa`). Canonical advisories should expose PDF links as `references.kind == "attachment"` and affected packages should surface `primitives.semVer.exactValue` for single-version hits.
+4. If Akamai blocks direct fetches, set `feedser:sources:icscisa:proxyUri` to your allow-listed egress proxy and rerun the dry-run.
+
+## 4. Rotation & Incident Response
+
+- Review GovDelivery access quarterly. Rotate the personalised code whenever Ops changes the service mailbox password or membership.  
+- Revoking the subscription in GovDelivery invalidates the code immediately; update the vault and configuration in the same change.  
+- If the code leaks, remove the subscription (`https://public.govdelivery.com/accounts/USDHSCISA/subscriber/manage_preferences?code=<CODE>`), resubscribe, and distribute the new value via the vault.
+
+## 5. Offline Kit Handling
+
+Include the personalised code in `offline-kit/secrets/feedser/icscisa.env`:
+
+```
+FEEDSER_SOURCES_ICSCISA_GOVDELIVERY_CODE=AB12CD34EF
+```
+
+The Offline Kit deployment script copies this file into the container secret directory mounted at `/run/secrets/feedser`. Ensure permissions are `600` and ownership matches the Feedser runtime user.
+
+## 6. Telemetry & Monitoring
+
+The connector emits metrics under the meter `StellaOps.Feedser.Source.Ics.Cisa`. They allow operators to track Akamai fallbacks, detail enrichment health, and advisory fan-out.
+
+- `icscisa.fetch.*` – counters for `attempts`, `success`, `failures`, `not_modified`, and `fallbacks`, plus histogram `icscisa.fetch.documents` showing documents added per topic pull (tags: `feedser.source`, `icscisa.topic`).
+- `icscisa.parse.*` – counters for `success`/`failures` and histograms `icscisa.parse.advisories`, `icscisa.parse.attachments`, `icscisa.parse.detail_fetches` to monitor enrichment workload per feed document.
+- `icscisa.detail.*` – counters `success` / `failures` per advisory (tagged with `icscisa.advisory`) to alert when Akamai blocks detail pages.
+- `icscisa.map.*` – counters for `success`/`failures` and histograms `icscisa.map.references`, `icscisa.map.packages`, `icscisa.map.aliases` capturing canonical fan-out.
+
+Suggested alerts:
+
+- `increase(icscisa.fetch.failures_total[15m]) > 0` or `increase(icscisa.fetch.fallbacks_total[15m]) > 5` — sustained Akamai or proxy issues.
+- `increase(icscisa.detail.failures_total[30m]) > 0` — detail enrichment breaking (potential HTML layout change).
+- `histogram_quantile(0.95, rate(icscisa.map.references_bucket[1h]))` trending sharply higher — sudden advisory reference explosion worth investigating.
+- Keep an eye on shared HTTP metrics (`feedser.source.http.*{feedser.source="ics-cisa"}`) for request latency and retry patterns.
+
+## 6. Related Tasks
+
+- `FEEDCONN-ICSCISA-02-009` (GovDelivery credential onboarding) — completed once this runbook is followed and secrets are placed in the vault.
+- `FEEDCONN-ICSCISA-02-007` (document inventory) — archive the first successful RSS response and any attachment URL schema under `docs/artifacts/icscisa/`.
--- a/docs/ops/feedser-kisa-operations.md
+++ b/docs/ops/feedser-kisa-operations.md
@@ -0,0 +1,74 @@
+# Feedser KISA Connector Operations
+
+Operational guidance for the Korea Internet & Security Agency (KISA / KNVD) connector (`source:kisa:*`). Pair this with the engineering brief in `docs/dev/kisa_connector_notes.md`.
+
+## 1. Prerequisites
+
+- Outbound HTTPS (or mirrored cache) for `https://knvd.krcert.or.kr/`.
+- Connector options defined under `feedser:sources:kisa`:
+
+```yaml
+feedser:
+  sources:
+    kisa:
+      feedUri: "https://knvd.krcert.or.kr/rss/securityInfo.do"
+      detailApiUri: "https://knvd.krcert.or.kr/rssDetailData.do"
+      detailPageUri: "https://knvd.krcert.or.kr/detailDos.do"
+      maxAdvisoriesPerFetch: 10
+      requestDelay: "00:00:01"
+      failureBackoff: "00:05:00"
+```
+
+> Ensure the URIs stay absolute—Feedser adds the `feedUri`/`detailApiUri` hosts to the HttpClient allow-list automatically.
+
+## 2. Staging Smoke Test
+
+1. Restart the Feedser workers so the KISA options bind.
+2. Run a full connector cycle:
+   - CLI: `stella db jobs run source:kisa:fetch --and-then source:kisa:parse --and-then source:kisa:map`
+   - REST: `POST /jobs/run { "kind": "source:kisa:fetch", "chain": ["source:kisa:parse", "source:kisa:map"] }`
+3. Confirm telemetry (Meter `StellaOps.Feedser.Source.Kisa`):
+   - `kisa.feed.success`, `kisa.feed.items`
+   - `kisa.detail.success` / `.failures`
+   - `kisa.parse.success` / `.failures`
+   - `kisa.map.success` / `.failures`
+   - `kisa.cursor.updates`
+4. Inspect logs for structured entries:
+   - `KISA feed returned {ItemCount}`
+   - `KISA fetched detail for {Idx} … category={Category}`
+   - `KISA mapped advisory {AdvisoryId} (severity={Severity})`
+   - Absence of warnings such as `document missing GridFS payload`.
+5. Validate MongoDB state:
+   - `raw_documents.metadata` has `kisa.idx`, `kisa.category`, `kisa.title`.
+   - DTO store contains `schemaVersion="kisa.detail.v1"`.
+   - Advisories include aliases (`IDX`, CVE) and `language="ko"`.
+   - `source_states` entry for `kisa` shows recent `cursor.lastFetchAt`.
+
+## 3. Production Monitoring
+
+- **Dashboards** – Add the following Prometheus/OTEL expressions:
+  - `rate(kisa_feed_items_total[15m])` versus `rate(feedser_source_http_requests_total{feedser_source="kisa"}[15m])`
+  - `increase(kisa_detail_failures_total{reason!="empty-document"}[1h])` alert at `>0`
+  - `increase(kisa_parse_failures_total[1h])` for storage/JSON issues
+  - `increase(kisa_map_failures_total[1h])` to flag schema drift
+  - `increase(kisa_cursor_updates_total[6h]) == 0` during active windows → warn
+- **Alerts** – Page when `rate(kisa_feed_success_total[2h]) == 0` while other connectors are active; back off for maintenance windows announced on `https://knvd.krcert.or.kr/`.
+- **Logs** – Watch for repeated warnings (`document missing`, `DTO missing`) or errors with reason tags `HttpRequestException`, `download`, `parse`, `map`.
+
+## 4. Localisation Handling
+
+- Hangul categories (for example `취약점정보`) flow into telemetry tags (`category=…`) and logs. Dashboards must render UTF‑8 and avoid transliteration.
+- HTML content is sanitised before storage; translation teams can consume the `ContentHtml` field safely.
+- Advisory severity remains as provided by KISA (`High`, `Medium`, etc.). Map-level failures include the severity tag for filtering.
+
+## 5. Fixture & Regression Maintenance
+
+- Regression fixtures: `src/StellaOps.Feedser.Source.Kisa.Tests/Fixtures/kisa-feed.xml` and `kisa-detail.json`.
+- Refresh via `UPDATE_KISA_FIXTURES=1 dotnet test src/StellaOps.Feedser.Source.Kisa.Tests/StellaOps.Feedser.Source.Kisa.Tests.csproj`.
+- The telemetry regression (`KisaConnectorTests.Telemetry_RecordsMetrics`) will fail if counters/log wiring drifts—treat failures as gating.
+
+## 6. Known Issues
+
+- RSS feeds only expose the latest 10 advisories; long outages require replay via archived feeds or manual IDX seeds.
+- Detail endpoint occasionally throttles; the connector honours `requestDelay` and reports failures with reason `HttpRequestException`. Consider increasing delay for weekend backfills.
+- If `kisa.category` tags suddenly appear as `unknown`, verify KISA has not renamed RSS elements; update the parser fixtures before production rollout.
--- a/docs/ops/feedser-msrc-operations.md
+++ b/docs/ops/feedser-msrc-operations.md
@@ -0,0 +1,86 @@
+# Feedser MSRC Connector – Azure AD Onboarding Brief
+
+_Drafted: 2025-10-15_
+
+## 1. App registration requirements
+
+- **Tenant**: shared StellaOps production Azure AD.
+- **Application type**: confidential client (web/API) issuing client credentials.
+- **API permissions**: `api://api.msrc.microsoft.com/.default` (Application). Admin consent required once.
+- **Token audience**: `https://api.msrc.microsoft.com/`.
+- **Grant type**: client credentials. Feedser will request tokens via `POST https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token`.
+
+## 2. Secret/credential policy
+
+- Maintain two client secrets (primary + standby) rotating every 90 days.
+- Store secrets in the Feedser secrets vault; Offline Kit deployments must mirror the secret payloads in their encrypted store.
+- Record rotation cadence in Ops runbook and update Feedser configuration (`FEEDSER__SOURCES__VNDR__MSRC__CLIENTSECRET`) ahead of expiry.
+
+## 3. Feedser configuration sample
+
+```yaml
+feedser:
+  sources:
+    vndr.msrc:
+      tenantId: "<azure-tenant-guid>"
+      clientId: "<app-registration-client-id>"
+      clientSecret: "<pull from secret store>"
+      apiVersion: "2024-08-01"
+      locale: "en-US"
+      requestDelay: "00:00:00.250"
+      failureBackoff: "00:05:00"
+      cursorOverlapMinutes: 10
+      downloadCvrf: false  # set true to persist CVRF ZIP alongside JSON detail
+```
+
+## 4. CVRF artefacts
+
+- The MSRC REST payload exposes `cvrfUrl` per advisory. Current connector persists the link as advisory metadata and reference; it does **not** download the ZIP by default.
+- Ops should mirror CVRF ZIPs when preparing Offline Kits so air-gapped deployments can reconcile advisories without direct internet access.
+- Once Offline Kit storage guidelines are finalised, extend the connector configuration with `downloadCvrf: true` to enable automatic attachment retrieval.
+
+### 4.1 State seeding helper
+
+Use `tools/SourceStateSeeder` to queue historical advisories (detail JSON + optional CVRF artefacts) for replay without manual Mongo edits. Example seed file:
+
+```json
+{
+  "source": "vndr.msrc",
+  "cursor": {
+    "lastModifiedCursor": "2024-01-01T00:00:00Z"
+  },
+  "documents": [
+    {
+      "uri": "https://api.msrc.microsoft.com/sug/v2.0/vulnerability/ADV2024-0001",
+      "contentFile": "./seeds/adv2024-0001.json",
+      "contentType": "application/json",
+      "metadata": { "msrc.vulnerabilityId": "ADV2024-0001" },
+      "addToPendingDocuments": true
+    },
+    {
+      "uri": "https://download.microsoft.com/msrc/2024/ADV2024-0001.cvrf.zip",
+      "contentFile": "./seeds/adv2024-0001.cvrf.zip",
+      "contentType": "application/zip",
+      "status": "mapped",
+      "addToPendingDocuments": false
+    }
+  ]
+}
+```
+
+Run the helper:
+
+```bash
+dotnet run --project tools/SourceStateSeeder -- \
+  --connection-string "mongodb://localhost:27017" \
+  --database feedser \
+  --input seeds/msrc-backfill.json
+```
+
+Any documents marked `addToPendingDocuments` will appear in the connector cursor; `DownloadCvrf` can remain disabled if the ZIP artefact is pre-seeded.
+
+## 5. Outstanding items
+
+- Ops to confirm tenant/app names and provide client credentials through the secure channel.
+- Connector team monitors token cache health (already implemented); validate instrumentation once Ops supplies credentials.
+- Offline Kit packaging: add encrypted blob containing client credentials with rotation instructions.
--- a/docs/ops/feedser-nkcki-operations.md
+++ b/docs/ops/feedser-nkcki-operations.md
@@ -0,0 +1,48 @@
+# NKCKI Connector Operations Guide
+
+## Overview
+
+The NKCKI connector ingests JSON bulletin archives from cert.gov.ru, expanding each `*.json.zip` attachment into per-vulnerability DTOs before canonical mapping. The fetch pipeline now supports cache-backed recovery, deterministic pagination, and telemetry suitable for production monitoring.
+
+## Configuration
+
+Key options exposed through `feedser:sources:ru-nkcki:http`:
+
+- `maxBulletinsPerFetch` – limits new bulletin downloads in a single run (default `5`).
+- `maxListingPagesPerFetch` – maximum listing pages visited during pagination (default `3`).
+- `listingCacheDuration` – minimum interval between listing fetches before falling back to cached artefacts (default `00:10:00`).
+- `cacheDirectory` – optional path for persisted bulletin archives used during offline or failure scenarios.
+- `requestDelay` – delay inserted between bulletin downloads to respect upstream politeness.
+
+When operating in offline-first mode, set `cacheDirectory` to a writable path (e.g. `/var/lib/feedser/cache/ru-nkcki`) and pre-populate bulletin archives via the offline kit.
+
+## Telemetry
+
+`RuNkckiDiagnostics` emits the following metrics under meter `StellaOps.Feedser.Source.Ru.Nkcki`:
+
+- `nkcki.listing.fetch.attempts` / `nkcki.listing.fetch.success` / `nkcki.listing.fetch.failures`
+- `nkcki.listing.pages.visited` (histogram, `pages`)
+- `nkcki.listing.attachments.discovered` / `nkcki.listing.attachments.new`
+- `nkcki.bulletin.fetch.success` / `nkcki.bulletin.fetch.cached` / `nkcki.bulletin.fetch.failures`
+- `nkcki.entries.processed` (histogram, `entries`)
+
+Integrate these counters into standard Feedser observability dashboards to track crawl coverage and cache hit rates.
+
+## Archive Backfill Strategy
+
+Bitrix pagination surfaces archives via `?PAGEN_1=n`. The connector now walks up to `maxListingPagesPerFetch` pages, deduplicating bulletin IDs and maintaining a rolling `knownBulletins` window. Backfill strategy:
+
+1. Enumerate pages from newest to oldest, respecting `maxListingPagesPerFetch` and `listingCacheDuration` to avoid refetch storms.
+2. Persist every `*.json.zip` attachment to the configured cache directory. This enables replay when listing access is temporarily blocked.
+3. During archive replay, `ProcessCachedBulletinsAsync` enqueues missing documents while respecting `maxVulnerabilitiesPerFetch`.
+4. For historical HTML-only advisories, collect page URLs and metadata while offline (future work: HTML and PDF extraction pipeline documented in `docs/feedser-connector-research-20251011.md`).
+
+For large migrations, seed caches with archived zip bundles, then run fetch/parse/map cycles in chronological order to maintain deterministic outputs.
+
+## Failure Handling
+
+- Listing failures mark the source state with exponential backoff while attempting cache replay.
+- Bulletin fetches fall back to cached copies before surfacing an error.
+- Mongo integration tests rely on bundled OpenSSL 1.1 libraries (`tools/openssl/linux-x64`) to keep `Mongo2Go` operational on modern distros.
+
+Refer to `ru-nkcki` entries in `src/StellaOps.Feedser.Source.Ru.Nkcki/TASKS.md` for outstanding items.