feat(concelier): multi-sprint batch (mirror domain + advisory sources + durable runtime + credentials)
Bundled commit covering pre-session work from multiple Concelier sprints already archived or in-flight: - SPRINT_20260419_006: mirror domain / source key validation - SPRINT_20260419_029 / 030: durable jobs orchestrator runtime + endpoint verification - SPRINT_20260421_001: advisory source projection truthful counts - SPRINT_20260421_002: FE advisory source consistency (connector-side bits) - SPRINT_20260421_003: advisory connector runtime alignment - SPRINT_20260422_003: source credential entry paths (in-flight) Includes connector internals (ACSC / Adobe / CERT-BUND / Chromium / Cisco / CVE-KEV / GHSA / JVN / KISA / MSRC / Oracle / Ubuntu), source management endpoints, mirror domain management, federation endpoints, topology setup, job registration, and associated dossier updates under docs/modules/concelier/. This commit groups ~229 file changes that accumulated across the above sprints; individual changes are preserved at file granularity so blame remains useful. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
master
@@ -3,7 +3,7 @@
|
||||
_Last updated: 2026-01-16_
|
||||
|
||||
## 1. Overview
|
||||
The ACSC connector ingests Australian Cyber Security Centre advisories and maps them to canonical IDs.
|
||||
The ACSC connector ingests Australian Cyber Security Centre advisories and maps them to canonical IDs. The runtime/catalog source ID is `auscert`.
|
||||
|
||||
## 2. Authentication
|
||||
- No authentication required for public feeds.
|
||||
@@ -12,13 +12,15 @@ The ACSC connector ingests Australian Cyber Security Centre advisories and maps
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
acsc:
|
||||
auscert:
|
||||
baseUri: "<acsc-feed-base>"
|
||||
maxDocumentsPerFetch: 20
|
||||
fetchTimeout: "00:00:45"
|
||||
requestDelay: "00:00:00"
|
||||
```
|
||||
|
||||
Legacy `concelier:sources:acsc` binding is still accepted for compatibility, but new setups should use `concelier:sources:auscert` so the documented defaults match `/api/v1/advisory-sources`.
|
||||
|
||||
## 4. Offline and air-gapped deployments
|
||||
- Mirror ACSC feeds into the Offline Kit and repoint `baseUri` to the mirror.
|
||||
|
||||
|
||||
@@ -1,26 +1,49 @@
|
||||
# Concelier Adobe PSIRT Connector - Operations Runbook
|
||||
|
||||
_Last updated: 2026-01-16_
|
||||
_Last updated: 2026-04-22_
|
||||
|
||||
## 1. Overview
|
||||
The Adobe connector ingests Adobe PSIRT advisories and maps them to canonical IDs.
|
||||
|
||||
The Adobe connector ingests Adobe PSIRT advisories and maps them to canonical IDs. The canonical runtime source ID is `adobe`.
|
||||
|
||||
## 2. Authentication
|
||||
|
||||
- No authentication required for public advisories.
|
||||
|
||||
## 3. Configuration (`concelier.yaml`)
|
||||
## 3. Configuration paths
|
||||
|
||||
Primary operator path:
|
||||
|
||||
- Web UI: **Security Posture -> Configure Sources** or **Ops -> Operations -> Feeds & Airgap -> Configure Sources**
|
||||
- CLI:
|
||||
```bash
|
||||
stella db connectors configure adobe \
|
||||
--server https://concelier.example.internal \
|
||||
--set indexUri=https://mirror.example.internal/adobe/security-bulletin.html
|
||||
```
|
||||
|
||||
The Adobe connector does not require credentials. Use the UI/CLI configuration path only when overriding the canonical Adobe public index or supplying additional mirror or index URIs.
|
||||
|
||||
Compatibility fallback (`concelier.yaml`):
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
adobe:
|
||||
baseUri: "<adobe-psirt-feed-base>"
|
||||
maxDocumentsPerFetch: 20
|
||||
fetchTimeout: "00:00:45"
|
||||
requestDelay: "00:00:00"
|
||||
indexUri: "https://helpx.adobe.com/security/security-bulletin.html"
|
||||
additionalIndexUris: []
|
||||
initialBackfill: "90.00:00:00"
|
||||
windowOverlap: "3.00:00:00"
|
||||
maxEntriesPerFetch: 100
|
||||
```
|
||||
|
||||
## 4. Offline and air-gapped deployments
|
||||
- Mirror advisories into the Offline Kit and repoint `baseUri` to the mirror.
|
||||
|
||||
- Mirror the Adobe index and detail pages into the Offline Kit.
|
||||
- Repoint `indexUri` and `additionalIndexUris` to the mirrored allowlisted endpoints.
|
||||
|
||||
## 5. Common failure modes
|
||||
- Upstream format changes or delayed bulletin updates.
|
||||
|
||||
- Adobe changes the bulletin index HTML layout or detail-table structure
|
||||
- Historic bulletin pages disappear or move without redirects
|
||||
- Operators mirror detail pages but forget to mirror the index page that seeds discovery
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
# Concelier CERT-Bund Connector Operations
|
||||
|
||||
_Last updated: 2025-10-17_
|
||||
_Last updated: 2025-10-17_
|
||||
|
||||
Runtime/catalog source ID: `cert-de`. Legacy `cert-bund` remains a compatibility-only configuration alias.
|
||||
|
||||
Germany’s Federal Office for Information Security (BSI) operates the Warn- und Informationsdienst (WID) portal. The Concelier CERT-Bund connector (`source:cert-bund:*`) ingests the public RSS feed, hydrates the portal’s JSON detail endpoint, and maps the result into canonical advisories while preserving the original German content.
|
||||
Germany’s Federal Office for Information Security (BSI) operates the Warn- und Informationsdienst (WID) portal. The Concelier CERT-Bund connector uses the runtime/catalog source ID `cert-de`, so the live fetch pipeline runs as `source:cert-de:*` while preserving the original German content.
|
||||
|
||||
---
|
||||
|
||||
@@ -17,12 +19,12 @@ Germany’s Federal Office for Information Security (BSI) operates the Warn- und
|
||||
Example `concelier.yaml` fragment:
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
cert-bund:
|
||||
feedUri: "https://wid.cert-bund.de/content/public/securityAdvisory/rss"
|
||||
portalBootstrapUri: "https://wid.cert-bund.de/portal/"
|
||||
detailApiUri: "https://wid.cert-bund.de/portal/api/securityadvisory"
|
||||
concelier:
|
||||
sources:
|
||||
cert-de:
|
||||
feedUri: "https://wid.cert-bund.de/content/public/securityAdvisory/rss"
|
||||
portalBootstrapUri: "https://wid.cert-bund.de/portal/"
|
||||
detailApiUri: "https://wid.cert-bund.de/portal/api/securityadvisory"
|
||||
maxAdvisoriesPerFetch: 50
|
||||
maxKnownAdvisories: 512
|
||||
requestTimeout: "00:00:30"
|
||||
@@ -59,7 +61,7 @@ Alerting ideas:
|
||||
|
||||
1. `increase(certbund.detail.fetch.failures_total[10m]) > 0`
|
||||
2. `rate(certbund.map.success_total[30m]) == 0`
|
||||
3. `histogram_quantile(0.95, rate(concelier_source_http_duration_bucket{concelier_source="cert-bund"}[15m])) > 5s`
|
||||
3. `histogram_quantile(0.95, rate(concelier_source_http_duration_bucket{concelier_source="cert-de"}[15m])) > 5s`
|
||||
|
||||
The WebService now registers the meter so metrics surface automatically once OpenTelemetry metrics are enabled.
|
||||
|
||||
@@ -124,7 +126,7 @@ operating offline.
|
||||
### 3.4 Connector-driven catch-up
|
||||
|
||||
1. Temporarily raise `maxAdvisoriesPerFetch` (e.g. 150) and reduce `requestDelay`.
|
||||
2. Run `stella db fetch --source cert-bund --stage fetch`, then `--stage parse`, then `--stage map` until the fetch log reports `enqueued=0`.
|
||||
2. Run `stella db fetch --source cert-de --stage fetch`, then `--stage parse`, then `--stage map` until the fetch log reports `enqueued=0`.
|
||||
3. Restore defaults and capture the cursor snapshot for audit.
|
||||
|
||||
---
|
||||
|
||||
@@ -1,26 +1,49 @@
|
||||
# Concelier Chromium Connector - Operations Runbook
|
||||
|
||||
_Last updated: 2026-01-16_
|
||||
_Last updated: 2026-04-22_
|
||||
|
||||
## 1. Overview
|
||||
The Chromium connector ingests Chromium security advisories and maps them to canonical IDs.
|
||||
|
||||
The Chromium connector ingests Chromium security advisories and maps them to canonical IDs. The canonical runtime source ID is `chromium`.
|
||||
|
||||
## 2. Authentication
|
||||
|
||||
- No authentication required for public advisories.
|
||||
|
||||
## 3. Configuration (`concelier.yaml`)
|
||||
## 3. Configuration paths
|
||||
|
||||
Primary operator path:
|
||||
|
||||
- Web UI: **Security Posture -> Configure Sources** or **Ops -> Operations -> Feeds & Airgap -> Configure Sources**
|
||||
- CLI:
|
||||
```bash
|
||||
stella db connectors configure chromium \
|
||||
--server https://concelier.example.internal \
|
||||
--set feedUri=https://mirror.example.internal/chromium/atom.xml
|
||||
```
|
||||
|
||||
The Chromium connector does not require credentials. Use the UI/CLI configuration path only when overriding the canonical Chrome Releases Atom feed for a mirror or controlled ingestion path.
|
||||
|
||||
Compatibility fallback (`concelier.yaml`):
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
chromium:
|
||||
baseUri: "<chromium-advisory-base>"
|
||||
maxDocumentsPerFetch: 20
|
||||
fetchTimeout: "00:00:45"
|
||||
requestDelay: "00:00:00"
|
||||
feedUri: "https://chromereleases.googleblog.com/atom.xml"
|
||||
initialBackfill: "30.00:00:00"
|
||||
windowOverlap: "2.00:00:00"
|
||||
maxFeedPages: 4
|
||||
maxEntriesPerPage: 50
|
||||
```
|
||||
|
||||
## 4. Offline and air-gapped deployments
|
||||
- Mirror advisories into the Offline Kit and repoint `baseUri` to the mirror.
|
||||
|
||||
- Mirror the Atom feed and referenced post pages into the Offline Kit.
|
||||
- Repoint `feedUri` to the mirrored allowlisted endpoint.
|
||||
|
||||
## 5. Common failure modes
|
||||
- Feed cadence shifts during Chromium release trains.
|
||||
|
||||
- Feed cadence shifts during Chromium release trains
|
||||
- Google changes the Atom feed or post markup used for stable-channel parsing
|
||||
- Operators mirror post pages but not the Atom feed that seeds discovery
|
||||
|
||||
@@ -1,94 +1,99 @@
|
||||
# Concelier Cisco PSIRT Connector – OAuth Provisioning SOP
|
||||
|
||||
_Last updated: 2025-10-14_
|
||||
|
||||
## 1. Scope
|
||||
|
||||
This runbook describes how Ops provisions, rotates, and distributes Cisco PSIRT openVuln OAuth client credentials for the Concelier Cisco connector. It covers online and air-gapped (Offline Kit) environments, quota-aware execution, and escalation paths.
|
||||
|
||||
## 2. Prerequisites
|
||||
|
||||
- Active Cisco.com (CCO) account with access to the Cisco API Console.
|
||||
- Cisco PSIRT openVuln API entitlement (visible under “My Apps & Keys” once granted).citeturn3search0
|
||||
- Concelier configuration location (typically `/etc/stella/concelier.yaml` in production) or Offline Kit secret bundle staging directory.
|
||||
|
||||
## 3. Provisioning workflow
|
||||
|
||||
1. **Register the application**
|
||||
- Sign in at <https://apiconsole.cisco.com>.
|
||||
- Select **Register a New App** → Application Type: `Service`, Grant Type: `Client Credentials`, API: `Cisco PSIRT openVuln API`.citeturn3search0
|
||||
- Record the generated `clientId` and `clientSecret` in the Ops vault.
|
||||
2. **Verify token issuance**
|
||||
- Request an access token with:
|
||||
```bash
|
||||
curl -s https://id.cisco.com/oauth2/default/v1/token \
|
||||
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||
-d "grant_type=client_credentials" \
|
||||
-d "client_id=${CLIENT_ID}" \
|
||||
-d "client_secret=${CLIENT_SECRET}"
|
||||
```
|
||||
- Confirm HTTP 200 and an `expires_in` value of 3600 seconds (tokens live for one hour).citeturn3search0turn3search7
|
||||
- Preserve the response only long enough to validate syntax; do **not** persist tokens.
|
||||
3. **Authorize Concelier runtime**
|
||||
- Update `concelier:sources:cisco:auth` (or the module-specific secret template) with the stored credentials.
|
||||
- For Offline Kit delivery, export encrypted secrets into `offline-kit/secrets/cisco-openvuln.json` using the platform’s sealed secret format.
|
||||
4. **Connectivity validation**
|
||||
- From the Concelier control plane, run `stella db fetch --source vndr-cisco --stage fetch` (use staging or a controlled window).
|
||||
- Ensure the Source HTTP diagnostics record `Bearer` authorization headers and no 401/403 responses.
|
||||
|
||||
## 4. Rotation SOP
|
||||
|
||||
| Step | Owner | Notes |
|
||||
| --- | --- | --- |
|
||||
| 1. Schedule rotation | Ops (monthly board) | Rotate every 90 days or immediately after suspected credential exposure. |
|
||||
| 2. Create replacement app | Ops | Repeat §3.1 with “-next” suffix; verify token issuance. |
|
||||
| 3. Stage dual credentials | Ops + Concelier On-Call | Publish new credentials to secret store alongside current pair. |
|
||||
| 4. Cut over | Concelier On-Call | Restart connector workers during a low-traffic window (<10 min) to pick up the new secret. |
|
||||
| 5. Deactivate legacy app | Ops | Delete prior app in Cisco API Console once telemetry confirms successful fetch/parse cycles for 2 consecutive hours. |
|
||||
|
||||
**Automation hooks**
|
||||
- Rotation reminders are tracked in OpsRunbookOps board (`OPS-RUN-KEYS` swim lane); add checklist items for Concelier Cisco when opening a rotation task.
|
||||
- Use the secret management pipeline (`ops/secrets/rotate.sh --connector cisco`) to template vault updates; the script renders a redacted diff for audit.
|
||||
|
||||
## 5. Offline Kit packaging
|
||||
|
||||
1. Generate the credential bundle using the Offline Kit CLI:
|
||||
`offline-kit secrets add cisco-openvuln --client-id … --client-secret …`
|
||||
2. Store the encrypted payload under `offline-kit/secrets/cisco-openvuln.enc`.
|
||||
3. Distribute via the Offline Kit channel; update `offline-kit/MANIFEST.md` with the credential fingerprint (SHA256 of plaintext concatenated with metadata).
|
||||
4. Document validation steps for the receiving site (token request from an air-gapped relay or cached token mirror).
|
||||
|
||||
## 6. Quota and throttling guidance
|
||||
|
||||
- Cisco enforces combined limits of 5 requests/second, 30 requests/minute, and 5 000 requests/day per application.citeturn0search0turn3search6
|
||||
- Concelier fetch jobs must respect `Retry-After` headers on HTTP 429 responses; Ops should monitor for sustained quota saturation and consider paging window adjustments.
|
||||
- Telemetry to watch: `concelier.source.http.requests{concelier.source="vndr-cisco"}`, `concelier.source.http.failures{...}`, and connector-specific metrics once implemented.
|
||||
|
||||
## 7. Telemetry & Monitoring
|
||||
|
||||
- **Metrics (Meter `StellaOps.Concelier.Connector.Vndr.Cisco`)**
|
||||
- `cisco.fetch.documents`, `cisco.fetch.failures`, `cisco.fetch.unchanged`
|
||||
- `cisco.parse.success`, `cisco.parse.failures`
|
||||
- `cisco.map.success`, `cisco.map.failures`, `cisco.map.affected.packages`
|
||||
- **Shared HTTP metrics** via `SourceDiagnostics`:
|
||||
- `concelier.source.http.requests{concelier.source="vndr-cisco"}`
|
||||
- `concelier.source.http.failures{concelier.source="vndr-cisco"}`
|
||||
- `concelier.source.http.duration{concelier.source="vndr-cisco"}`
|
||||
- **Structured logs**
|
||||
- `Cisco fetch completed date=… pages=… added=…` (info)
|
||||
- `Cisco parse completed parsed=… failures=…` (info)
|
||||
- `Cisco map completed mapped=… failures=…` (info)
|
||||
- Warnings surface when DTO serialization fails or document payload is missing.
|
||||
- Suggested alerts: non-zero `cisco.fetch.failures` in 15m, or `cisco.map.success` flatlines while fetch continues.
|
||||
|
||||
## 8. Incident response
|
||||
|
||||
- **Token compromise** – revoke the application in the Cisco API Console, purge cached secrets, rotate immediately per §4.
|
||||
- **Persistent 401/403** – confirm credentials in vault, then validate token issuance; if unresolved, open a Cisco DevNet support ticket referencing the application ID.
|
||||
- **429 spikes** – inspect job scheduler cadence and adjust connector options (`maxRequestsPerWindow`) before requesting higher quotas from Cisco.
|
||||
|
||||
## 9. References
|
||||
|
||||
- Cisco PSIRT openVuln API Authentication Guide.citeturn3search0
|
||||
- Accessing the openVuln API using curl (token lifetime).citeturn3search7
|
||||
- openVuln API rate limit documentation.citeturn0search0turn3search6
|
||||
# Concelier Cisco PSIRT Connector - OAuth Provisioning SOP
|
||||
|
||||
_Last updated: 2026-04-22_
|
||||
|
||||
## 0. Scope note
|
||||
|
||||
This runbook covers the authenticated Concelier Cisco openVuln advisory connector.
|
||||
|
||||
It does not describe the default Excititor public VEX mirror bootstrap. The default VEX bootstrap path remains credential-free and uses Cisco public CSAF metadata and documents.
|
||||
|
||||
## 1. Scope
|
||||
|
||||
This runbook describes how Ops provisions, rotates, and distributes Cisco PSIRT openVuln OAuth client credentials for the Concelier Cisco connector. It covers online and air-gapped environments, quota-aware execution, and escalation paths.
|
||||
|
||||
Primary operator path:
|
||||
|
||||
- Web UI: **Security Posture -> Configure Sources** or **Ops -> Operations -> Feeds & Airgap -> Configure Sources**
|
||||
- CLI:
|
||||
```bash
|
||||
stella db connectors configure cisco \
|
||||
--server https://concelier.example.internal \
|
||||
--set clientId=... \
|
||||
--set clientSecret=...
|
||||
```
|
||||
|
||||
Persisted source configuration is now the preferred operator path. Host-local YAML and external secret distribution remain compatibility fallbacks for older installs.
|
||||
|
||||
## 2. Prerequisites
|
||||
|
||||
- Active Cisco.com account with access to the Cisco API Console
|
||||
- Cisco PSIRT openVuln API entitlement visible under **My Apps & Keys**
|
||||
- Concelier control-plane access
|
||||
|
||||
Entitlement notes:
|
||||
|
||||
- There is no separate StellaOps-side fee for this connector.
|
||||
- Cisco still requires a Cisco.com account, API Console registration, and visible access to the PSIRT openVuln API entry.
|
||||
- If the openVuln API is not visible for the account, the connector cannot be fully activated until Cisco-side entitlement is in place.
|
||||
|
||||
## 3. Provisioning workflow
|
||||
|
||||
1. Register the application
|
||||
- Sign in at `https://apiconsole.cisco.com`.
|
||||
- Open **My Apps & Keys**.
|
||||
- Click **Register a New App**.
|
||||
- Choose:
|
||||
- Application Type: `Service`
|
||||
- Grant Type: `Client Credentials`
|
||||
- API: `Cisco PSIRT openVuln API`
|
||||
- Record the generated `clientId` and `clientSecret`.
|
||||
2. Verify token issuance
|
||||
- Request an access token:
|
||||
```bash
|
||||
curl -s https://id.cisco.com/oauth2/default/v1/token \
|
||||
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||
-d "grant_type=client_credentials" \
|
||||
-d "client_id=${CLIENT_ID}" \
|
||||
-d "client_secret=${CLIENT_SECRET}"
|
||||
```
|
||||
- Confirm HTTP 200 and a usable `expires_in` value.
|
||||
- Do not persist the token itself.
|
||||
3. Authorize Concelier runtime
|
||||
- Store the `clientId` and `clientSecret` in StellaOps through the Web UI or the CLI command shown above.
|
||||
4. Connectivity validation
|
||||
- From the Concelier control plane, run:
|
||||
```bash
|
||||
stella db fetch --source cisco --stage fetch
|
||||
```
|
||||
- Confirm no 401/403 responses in source diagnostics.
|
||||
|
||||
## 4. Rotation SOP
|
||||
|
||||
| Step | Owner | Notes |
|
||||
| --- | --- | --- |
|
||||
| 1. Schedule rotation | Ops | Rotate every 90 days or immediately after suspected exposure. |
|
||||
| 2. Create replacement app | Ops | Repeat the registration flow and verify token issuance. |
|
||||
| 3. Stage new credentials | Ops + Concelier on-call | Store the replacement values in StellaOps and verify a staged fetch. |
|
||||
| 4. Cut over | Concelier on-call | Validate successful fetch/parse cycles during a low-traffic window. |
|
||||
| 5. Deactivate legacy app | Ops | Delete the old app in Cisco API Console after successful validation. |
|
||||
|
||||
## 5. Offline Kit packaging
|
||||
|
||||
If the site still mirrors secrets out of band:
|
||||
|
||||
1. Generate the credential bundle using the Offline Kit CLI.
|
||||
2. Store the encrypted payload under the site-approved secret bundle location.
|
||||
3. Record the delivery fingerprint in the manifest.
|
||||
|
||||
## 6. Quota and throttling guidance
|
||||
|
||||
- Cisco documents combined limits for requests per second, minute, and day per application.
|
||||
- Concelier must respect `Retry-After` on HTTP 429 responses.
|
||||
- If quota saturation persists, adjust job cadence before requesting higher quotas from Cisco.
|
||||
|
||||
## 7. Incident response
|
||||
|
||||
- Token compromise: revoke the application in Cisco API Console, purge retained credentials, rotate immediately.
|
||||
- Persistent 401/403: verify the stored credentials, then verify token issuance directly against Cisco.
|
||||
- 429 spikes: inspect scheduler cadence and source tuning before escalating upstream.
|
||||
|
||||
@@ -115,9 +115,32 @@ Treat repeated schema failures or growing anomaly counts as an upstream regressi
|
||||
- `kev.map.advisories` (tag `catalogVersion`)
|
||||
4. Confirm `concelier.source.http.requests_total{concelier_source="kev"}` increments once per fetch and that the paired `concelier.source.http.failures_total` stays flat (zero increase).
|
||||
5. Inspect the info logs `Fetched KEV catalog document … pendingDocuments=…` and `Parsed KEV catalog document … entries=…`—they should appear exactly once per run and `Mapped X/Y… skipped=0` should match the `kev.map.advisories` delta.
|
||||
6. Confirm PostgreSQL records exist for the catalog JSON (`raw_documents` and `dtos` tables in schema `vuln`) and that advisories with prefix `kev/` are written.
|
||||
|
||||
### 2.4 Production Monitoring
|
||||
6. Confirm PostgreSQL records exist for the catalog JSON (`raw_documents` and `dtos` tables in schema `vuln`) and that advisories with prefix `kev/` are written.
|
||||
|
||||
### 2.3.1 Cached Replay Recovery
|
||||
|
||||
Use this recovery when KEV fetches already succeeded against an unchanged upstream catalog, but a mapper or persistence fix needs to rebuild `source_id` bindings or canonical/source-edge projections from the latest cached catalog.
|
||||
|
||||
1. Trigger a forced parse replay against the latest cached KEV document:
|
||||
- `POST /jobs/source:kev:parse`
|
||||
- Body: `{"trigger":"api","parameters":{"force":"true"}}`
|
||||
2. Wait for the parse run to finish with `status=Succeeded` via `GET /jobs/{runId}`.
|
||||
3. Trigger a forced map replay against the same cached document set:
|
||||
- `POST /jobs/source:kev:map`
|
||||
- Body: `{"trigger":"api","parameters":{"force":"true"}}`
|
||||
4. Wait for the map run to finish with `status=Succeeded` via `GET /jobs/{runId}`.
|
||||
5. Recheck `GET /api/v1/advisory-sources/kev/freshness` and confirm the KEV rollups are repopulated:
|
||||
- `sourceDocumentCount > 0`
|
||||
- `canonicalAdvisoryCount > 0`
|
||||
- `cveCount > 0`
|
||||
- `totalAdvisories > 0`
|
||||
6. Verify PostgreSQL recovery state:
|
||||
- `vuln.advisories` rows with advisory keys prefixed `kev/` have non-null `source_id`
|
||||
- `vuln.advisory_source_edge` contains rows for the KEV source
|
||||
|
||||
This path reuses the most recent cached KEV catalog and does not wait for CISA to publish a newer document. It is the required operator recovery step after shipping KEV mapper or persistence fixes onto a host that already cached the current catalog.
|
||||
|
||||
### 2.4 Production Monitoring
|
||||
|
||||
- Alert when `rate(kev_fetch_success_total[8h]) == 0` during working hours (daily cadence breach) and when `increase(kev_fetch_failures_total[1h]) > 0`.
|
||||
- Page the on-call if `increase(kev_parse_failures_total{reason="schema"}[6h]) > 0`—this usually signals an upstream payload change. Treat repeated `reason="download"` spikes as networking issues to the mirror.
|
||||
|
||||
@@ -1,123 +1,103 @@
|
||||
# Concelier GHSA Connector – Operations Runbook
|
||||
|
||||
_Last updated: 2025-10-16_
|
||||
|
||||
## 1. Overview
|
||||
The GitHub Security Advisories (GHSA) connector pulls advisory metadata from the GitHub REST API `/security/advisories` endpoint. GitHub enforces both primary and secondary rate limits, so operators must monitor usage and configure retries to avoid throttling incidents.
|
||||
|
||||
## 2. Rate-limit telemetry
|
||||
The connector now surfaces rate-limit headers on every fetch and exposes the following metrics via OpenTelemetry:
|
||||
|
||||
| Metric | Description | Tags |
|
||||
|--------|-------------|------|
|
||||
| `ghsa.ratelimit.limit` (histogram) | Samples the reported request quota at fetch time. | `phase` = `list` or `detail`, `resource` (e.g., `core`). |
|
||||
| `ghsa.ratelimit.remaining` (histogram) | Remaining requests returned by `X-RateLimit-Remaining`. | `phase`, `resource`. |
|
||||
| `ghsa.ratelimit.reset_seconds` (histogram) | Seconds until `X-RateLimit-Reset`. | `phase`, `resource`. |
|
||||
| `ghsa.ratelimit.headroom_pct` (histogram) | Percentage of the quota still available (`remaining / limit * 100`). | `phase`, `resource`. |
|
||||
| `ghsa.ratelimit.headroom_pct_current` (observable gauge) | Latest headroom percentage reported per resource. | `phase`, `resource`. |
|
||||
| `ghsa.ratelimit.exhausted` (counter) | Incremented whenever GitHub returns a zero remaining quota and the connector delays before retrying. | `phase`. |
|
||||
|
||||
### Dashboards & alerts
|
||||
- Plot `ghsa.ratelimit.remaining` as the latest value to watch the runway. Alert when the value stays below **`RateLimitWarningThreshold`** (default `500`) for more than 5 minutes.
|
||||
- Use `ghsa.ratelimit.headroom_pct_current` to visualise remaining quota % — paging once it sits below **10 %** for longer than a single reset window helps avoid secondary limits.
|
||||
- Raise a separate alert on `increase(ghsa.ratelimit.exhausted[15m]) > 0` to catch hard throttles.
|
||||
- Overlay `ghsa.fetch.attempts` vs `ghsa.fetch.failures` to confirm retries are effective.
|
||||
|
||||
## 3. Logging signals
|
||||
When `X-RateLimit-Remaining` falls below `RateLimitWarningThreshold`, the connector emits:
|
||||
```
|
||||
GHSA rate limit warning: remaining {Remaining}/{Limit} for {Phase} {Resource} (headroom {Headroom}%)
|
||||
```
|
||||
When GitHub reports zero remaining calls, the connector logs and sleeps for the reported `Retry-After`/`X-RateLimit-Reset` interval (falling back to `SecondaryRateLimitBackoff`).
|
||||
|
||||
After the quota recovers above the warning threshold the connector writes an informational log with the refreshed remaining/headroom, letting operators clear alerts quickly.
|
||||
|
||||
## 4. Configuration knobs (`concelier.yaml`)
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
ghsa:
|
||||
apiToken: "${GITHUB_PAT}"
|
||||
pageSize: 50
|
||||
requestDelay: "00:00:00.200"
|
||||
failureBackoff: "00:05:00"
|
||||
rateLimitWarningThreshold: 500 # warn below this many remaining calls
|
||||
secondaryRateLimitBackoff: "00:02:00" # fallback delay when GitHub omits Retry-After
|
||||
```
|
||||
|
||||
### Recommendations
|
||||
- Increase `requestDelay` in air-gapped or burst-heavy deployments to smooth token consumption.
|
||||
- Lower `rateLimitWarningThreshold` only if your dashboards already page on the new histogram; never set it negative.
|
||||
- For bots using a low-privilege PAT, keep `secondaryRateLimitBackoff` at ≥60 seconds to respect GitHub’s secondary-limit guidance.
|
||||
|
||||
#### Default job schedule
|
||||
|
||||
| Job kind | Cron | Timeout | Lease |
|
||||
|----------|------|---------|-------|
|
||||
| `source:ghsa:fetch` | `1,11,21,31,41,51 * * * *` | 6 minutes | 4 minutes |
|
||||
| `source:ghsa:parse` | `3,13,23,33,43,53 * * * *` | 5 minutes | 4 minutes |
|
||||
| `source:ghsa:map` | `5,15,25,35,45,55 * * * *` | 5 minutes | 4 minutes |
|
||||
|
||||
These defaults spread GHSA stages across the hour so fetch completes before parse/map fire. Override them via `concelier.jobs.definitions[...]` when coordinating multiple connectors on the same runner.
|
||||
|
||||
## 5. Provisioning credentials
|
||||
|
||||
Concelier requires a GitHub personal access token (classic) with the **`read:org`** and **`security_events`** scopes to pull GHSA data. Store it as a secret and reference it via `concelier.sources.ghsa.apiToken`.
|
||||
|
||||
### Docker Compose (stack operators)
|
||||
```yaml
|
||||
services:
|
||||
concelier:
|
||||
environment:
|
||||
CONCELIER__SOURCES__GHSA__APITOKEN: /run/secrets/ghsa_pat
|
||||
secrets:
|
||||
- ghsa_pat
|
||||
|
||||
secrets:
|
||||
ghsa_pat:
|
||||
file: ./secrets/ghsa_pat.txt # contains only the PAT value
|
||||
```
|
||||
|
||||
### Helm values (cluster operators)
|
||||
```yaml
|
||||
concelier:
|
||||
extraEnv:
|
||||
- name: CONCELIER__SOURCES__GHSA__APITOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: concelier-ghsa
|
||||
key: apiToken
|
||||
|
||||
extraSecrets:
|
||||
concelier-ghsa:
|
||||
apiToken: "<paste PAT here or source from external secret store>"
|
||||
```
|
||||
|
||||
After rotating the PAT, restart the Concelier workers (or run `kubectl rollout restart deployment/concelier`) to ensure the configuration reloads.
|
||||
|
||||
When enabling GHSA the first time, run a staged backfill:
|
||||
|
||||
1. Trigger `source:ghsa:fetch` manually (CLI or API) outside of peak hours.
|
||||
2. Watch `concelier.jobs.health` for the GHSA jobs until they report `healthy`.
|
||||
3. Allow the scheduled cron cadence to resume once the initial backlog drains (typically < 30 minutes).
|
||||
|
||||
## 6. Runbook steps when throttled
|
||||
1. Check `ghsa.ratelimit.exhausted` for the affected phase (`list` vs `detail`).
|
||||
2. Confirm the connector is delaying—logs will show `GHSA rate limit exhausted...` with the chosen backoff.
|
||||
3. If rate limits stay exhausted:
|
||||
- Verify no other jobs are sharing the PAT.
|
||||
- Temporarily reduce `MaxPagesPerFetch` or `PageSize` to shrink burst size.
|
||||
- Consider provisioning a dedicated PAT (GHSA permissions only) for Concelier.
|
||||
4. After the quota resets, reset `rateLimitWarningThreshold`/`requestDelay` to their normal values and monitor the histograms for at least one hour.
|
||||
|
||||
## 7. Alert integration quick reference
|
||||
- Prometheus: `ghsa_ratelimit_remaining_bucket` (from histogram) – use `histogram_quantile(0.99, ...)` to trend capacity.
|
||||
- VictoriaMetrics: `LAST_over_time(ghsa_ratelimit_remaining_sum[5m])` for simple last-value graphs.
|
||||
- Grafana: stack remaining + used to visualise total limit per resource.
|
||||
|
||||
## 8. Canonical metric fallback analytics
|
||||
When GitHub omits CVSS vectors/scores, the connector now assigns a deterministic canonical metric id in the form `ghsa:severity/<level>` and publishes it to Merge so severity precedence still resolves against GHSA even without CVSS data.
|
||||
|
||||
- Metric: `ghsa.map.canonical_metric_fallbacks` (counter) with tags `severity`, `canonical_metric_id`, `reason=no_cvss`.
|
||||
- Monitor the counter alongside Merge parity checks; a sudden spike suggests GitHub is shipping advisories without vectors and warrants cross-checking downstream exporters.
|
||||
- Because the canonical id feeds Merge, parity dashboards should overlay this metric to confirm fallback advisories continue to merge ahead of downstream sources when GHSA supplies more recent data.
|
||||
# Concelier GHSA Connector - Operations Runbook
|
||||
|
||||
_Last updated: 2026-04-22_
|
||||
|
||||
## 1. Overview
|
||||
|
||||
The GitHub Security Advisories (GHSA) connector pulls advisory metadata from the GitHub REST API `/security/advisories` endpoint. GitHub enforces both primary and secondary rate limits, so operators must monitor usage and configure retries to avoid throttling incidents.
|
||||
|
||||
## 2. Rate-limit telemetry
|
||||
|
||||
The connector surfaces rate-limit headers on every fetch and exposes the following metrics via OpenTelemetry:
|
||||
|
||||
| Metric | Description | Tags |
|
||||
| --- | --- | --- |
|
||||
| `ghsa.ratelimit.limit` | Samples the reported request quota at fetch time. | `phase`, `resource` |
|
||||
| `ghsa.ratelimit.remaining` | Remaining requests returned by `X-RateLimit-Remaining`. | `phase`, `resource` |
|
||||
| `ghsa.ratelimit.reset_seconds` | Seconds until `X-RateLimit-Reset`. | `phase`, `resource` |
|
||||
| `ghsa.ratelimit.headroom_pct` | Percentage of quota still available. | `phase`, `resource` |
|
||||
| `ghsa.ratelimit.headroom_pct_current` | Latest headroom percentage per resource. | `phase`, `resource` |
|
||||
| `ghsa.ratelimit.exhausted` | Incremented whenever GitHub returns zero remaining quota and the connector delays before retrying. | `phase` |
|
||||
|
||||
Dashboards and alerts:
|
||||
|
||||
- Alert when `ghsa.ratelimit.remaining` stays below `RateLimitWarningThreshold` for more than 5 minutes.
|
||||
- Page if `ghsa.ratelimit.headroom_pct_current` remains below 10% for longer than a reset window.
|
||||
- Alert on `increase(ghsa.ratelimit.exhausted[15m]) > 0`.
|
||||
|
||||
## 3. Configuration paths
|
||||
|
||||
Primary operator path:
|
||||
|
||||
- Web UI: **Security Posture -> Configure Sources** or **Ops -> Operations -> Feeds & Airgap -> Configure Sources**
|
||||
- CLI:
|
||||
```bash
|
||||
stella db connectors configure ghsa \
|
||||
--server https://concelier.example.internal \
|
||||
--set apiToken=github_pat_xxx
|
||||
```
|
||||
|
||||
The persisted source configuration key is `apiToken`. Secrets are retained server-side and are not echoed back after storage.
|
||||
|
||||
Compatibility fallback (`concelier.yaml`):
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
ghsa:
|
||||
apiToken: "${GITHUB_PAT}"
|
||||
pageSize: 50
|
||||
requestDelay: "00:00:00.200"
|
||||
failureBackoff: "00:05:00"
|
||||
rateLimitWarningThreshold: 500
|
||||
secondaryRateLimitBackoff: "00:02:00"
|
||||
```
|
||||
|
||||
Recommendations:
|
||||
|
||||
- Increase `requestDelay` in burst-heavy or mirrored deployments.
|
||||
- Lower `rateLimitWarningThreshold` only if dashboards already page on the telemetry above.
|
||||
- Keep `secondaryRateLimitBackoff` at 60 seconds or more for low-privilege tokens.
|
||||
|
||||
## 4. Credential acquisition
|
||||
|
||||
Where to sign in:
|
||||
|
||||
- `https://github.com/settings/personal-access-tokens`
|
||||
|
||||
What to create:
|
||||
|
||||
- A GitHub personal access token or a GitHub App token that can call the security advisories REST endpoints
|
||||
|
||||
Upstream notes:
|
||||
|
||||
- GitHub documents that the global security advisories endpoint can serve public reviewed advisories anonymously and works with fine-grained PATs.
|
||||
- The current StellaOps GHSA connector still validates that `apiToken` is present, so skipping the token is not supported for this connector path.
|
||||
- If the target organization uses SAML SSO, authorize the PAT for that org after creation.
|
||||
- Some organizations restrict classic PATs or require PAT approval. That policy is external to StellaOps.
|
||||
|
||||
Legacy host/env distribution remains available for older deployments, but it is no longer the preferred operator path.
|
||||
|
||||
## 5. Default job schedule
|
||||
|
||||
| Job kind | Cron | Timeout | Lease |
|
||||
| --- | --- | --- | --- |
|
||||
| `source:ghsa:fetch` | `1,11,21,31,41,51 * * * *` | 6 minutes | 4 minutes |
|
||||
| `source:ghsa:parse` | `3,13,23,33,43,53 * * * *` | 5 minutes | 4 minutes |
|
||||
| `source:ghsa:map` | `5,15,25,35,45,55 * * * *` | 5 minutes | 4 minutes |
|
||||
|
||||
These defaults spread GHSA stages across the hour so fetch completes before parse/map fire.
|
||||
|
||||
## 6. Runbook steps when throttled
|
||||
|
||||
1. Check `ghsa.ratelimit.exhausted` for the affected phase.
|
||||
2. Confirm the connector is delaying before retry.
|
||||
3. If exhaustion persists:
|
||||
- Verify no unrelated jobs are sharing the token.
|
||||
- Temporarily reduce `MaxPagesPerFetch` or `PageSize`.
|
||||
- Consider provisioning a dedicated token.
|
||||
4. After quota resets, return settings to their normal values and monitor for at least one hour.
|
||||
|
||||
## 7. Canonical metric fallback analytics
|
||||
|
||||
When GitHub omits CVSS vectors or scores, the connector assigns a deterministic canonical metric ID in the form `ghsa:severity/<level>` and publishes it to Merge so severity precedence still resolves against GHSA even without CVSS data.
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
_Last updated: 2026-01-16_
|
||||
|
||||
## 1. Overview
|
||||
The JVN connector ingests Japan Vulnerability Notes (JVN) advisories and maps them to canonical IDs.
|
||||
The JVN connector ingests Japan Vulnerability Notes (JVN) advisories and maps them to canonical IDs. The runtime/catalog source ID is `jpcert`.
|
||||
|
||||
## 2. Authentication
|
||||
- No authentication required for public feeds.
|
||||
@@ -12,13 +12,15 @@ The JVN connector ingests Japan Vulnerability Notes (JVN) advisories and maps th
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
jvn:
|
||||
jpcert:
|
||||
baseUri: "<jvn-feed-base>"
|
||||
maxDocumentsPerFetch: 20
|
||||
fetchTimeout: "00:00:45"
|
||||
requestDelay: "00:00:00"
|
||||
```
|
||||
|
||||
Legacy `concelier:sources:jvn` binding is still accepted for compatibility, but operators should use `concelier:sources:jpcert` for new installs and UI-driven setup.
|
||||
|
||||
## 4. Offline and air-gapped deployments
|
||||
- Mirror JVN feeds into the Offline Kit and repoint `baseUri` to the mirror.
|
||||
|
||||
|
||||
@@ -1,19 +1,19 @@
|
||||
# Concelier KISA Connector Operations
|
||||
|
||||
Operational guidance for the Korea Internet & Security Agency (KISA / KNVD) connector (`source:kisa:*`). Pair this with the engineering brief in `docs/dev/kisa_connector_notes.md`.
|
||||
Operational guidance for the Korea Internet & Security Agency (KISA / KNVD) connector. The runtime/catalog source ID is `krcert`, so the live fetch pipeline runs as `source:krcert:*`. Pair this with the engineering brief in `docs/dev/kisa_connector_notes.md`.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Outbound HTTPS (or mirrored cache) for `https://knvd.krcert.or.kr/`.
|
||||
- Connector options defined under `concelier:sources:kisa`:
|
||||
- Connector options defined under `concelier:sources:krcert`:
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
kisa:
|
||||
feedUri: "https://knvd.krcert.or.kr/rss/securityInfo.do"
|
||||
detailApiUri: "https://knvd.krcert.or.kr/rssDetailData.do"
|
||||
detailPageUri: "https://knvd.krcert.or.kr/detailDos.do"
|
||||
krcert:
|
||||
feedUri: "https://knvd.krcert.or.kr/rss/securityInfo.do"
|
||||
detailApiUri: "https://knvd.krcert.or.kr/rssDetailData.do"
|
||||
detailPageUri: "https://knvd.krcert.or.kr/detailDos.do"
|
||||
maxAdvisoriesPerFetch: 10
|
||||
requestDelay: "00:00:01"
|
||||
failureBackoff: "00:05:00"
|
||||
@@ -21,12 +21,14 @@ concelier:
|
||||
|
||||
> Ensure the URIs stay absolute—Concelier adds the `feedUri`/`detailApiUri` hosts to the HttpClient allow-list automatically.
|
||||
|
||||
## 2. Staging Smoke Test
|
||||
Legacy `concelier:sources:kisa` binding is still accepted for compatibility, but new setups should use `concelier:sources:krcert`.
|
||||
|
||||
## 2. Staging Smoke Test
|
||||
|
||||
1. Restart the Concelier workers so the KISA options bind.
|
||||
2. Run a full connector cycle:
|
||||
- CLI: run `stella db fetch --source kisa --stage fetch`, then `--stage parse`, then `--stage map`.
|
||||
- REST: `POST /jobs/run { "kind": "source:kisa:fetch", "chain": ["source:kisa:parse", "source:kisa:map"] }`
|
||||
- CLI: run `stella db fetch --source krcert --stage fetch`, then `--stage parse`, then `--stage map`.
|
||||
- REST: `POST /jobs/run { "kind": "source:krcert:fetch", "chain": ["source:krcert:parse", "source:krcert:map"] }`
|
||||
3. Confirm telemetry (Meter `StellaOps.Concelier.Connector.Kisa`):
|
||||
- `kisa.feed.success`, `kisa.feed.items`
|
||||
- `kisa.detail.success` / `.failures`
|
||||
@@ -42,7 +44,9 @@ concelier:
|
||||
- `raw_documents` table metadata has `kisa.idx`, `kisa.category`, `kisa.title`.
|
||||
- `dtos` table contains `schemaVersion="kisa.detail.v1"`.
|
||||
- `advisories` table includes aliases (`IDX`, CVE) and `language="ko"`.
|
||||
- `source_states` entry for `kisa` shows recent `cursor.lastFetchAt`.
|
||||
- `source_states` entry for `krcert` shows recent `cursor.lastFetchAt`.
|
||||
|
||||
Telemetry and fixture naming still use the historical `kisa.*` metric namespace, so dashboards should keep those series names even though the runtime source ID is now `krcert`.
|
||||
|
||||
## 3. Production Monitoring
|
||||
|
||||
|
||||
@@ -1,86 +1,68 @@
|
||||
# Concelier MSRC Connector – Azure AD Onboarding Brief
|
||||
|
||||
_Drafted: 2025-10-15_
|
||||
|
||||
## 1. App registration requirements
|
||||
|
||||
- **Tenant**: shared StellaOps production Azure AD.
|
||||
- **Application type**: confidential client (web/API) issuing client credentials.
|
||||
- **API permissions**: `api://api.msrc.microsoft.com/.default` (Application). Admin consent required once.
|
||||
- **Token audience**: `https://api.msrc.microsoft.com/`.
|
||||
- **Grant type**: client credentials. Concelier will request tokens via `POST https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token`.
|
||||
|
||||
## 2. Secret/credential policy
|
||||
|
||||
- Maintain two client secrets (primary + standby) rotating every 90 days.
|
||||
- Store secrets in the Concelier secrets vault; Offline Kit deployments must mirror the secret payloads in their encrypted store.
|
||||
- Record rotation cadence in Ops runbook and update Concelier configuration (`CONCELIER__SOURCES__VNDR__MSRC__CLIENTSECRET`) ahead of expiry.
|
||||
|
||||
## 3. Concelier configuration sample
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
vndr.msrc:
|
||||
tenantId: "<azure-tenant-guid>"
|
||||
clientId: "<app-registration-client-id>"
|
||||
clientSecret: "<pull from secret store>"
|
||||
apiVersion: "2024-08-01"
|
||||
locale: "en-US"
|
||||
requestDelay: "00:00:00.250"
|
||||
failureBackoff: "00:05:00"
|
||||
cursorOverlapMinutes: 10
|
||||
downloadCvrf: false # set true to persist CVRF ZIP alongside JSON detail
|
||||
```
|
||||
|
||||
## 4. CVRF artefacts
|
||||
|
||||
- The MSRC REST payload exposes `cvrfUrl` per advisory. Current connector persists the link as advisory metadata and reference; it does **not** download the ZIP by default.
|
||||
- Ops should mirror CVRF ZIPs when preparing Offline Kits so air-gapped deployments can reconcile advisories without direct internet access.
|
||||
- Once Offline Kit storage guidelines are finalised, extend the connector configuration with `downloadCvrf: true` to enable automatic attachment retrieval.
|
||||
|
||||
### 4.1 State seeding helper
|
||||
|
||||
Use `src/Tools/SourceStateSeeder` to queue historical advisories (detail JSON + optional CVRF artefacts) for replay without manual database edits. Example seed file:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "vndr.msrc",
|
||||
"cursor": {
|
||||
"lastModifiedCursor": "2024-01-01T00:00:00Z"
|
||||
},
|
||||
"documents": [
|
||||
{
|
||||
"uri": "https://api.msrc.microsoft.com/sug/v2.0/vulnerability/ADV2024-0001",
|
||||
"contentFile": "./seeds/adv2024-0001.json",
|
||||
"contentType": "application/json",
|
||||
"metadata": { "msrc.vulnerabilityId": "ADV2024-0001" },
|
||||
"addToPendingDocuments": true
|
||||
},
|
||||
{
|
||||
"uri": "https://download.microsoft.com/msrc/2024/ADV2024-0001.cvrf.zip",
|
||||
"contentFile": "./seeds/adv2024-0001.cvrf.zip",
|
||||
"contentType": "application/zip",
|
||||
"status": "mapped",
|
||||
"addToPendingDocuments": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Run the helper:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Tools/SourceStateSeeder -- \
|
||||
--connection-string "Host=localhost;Database=stellaops;Username=stella;Password=..." \
|
||||
--schema vuln \
|
||||
--input seeds/msrc-backfill.json
|
||||
```
|
||||
|
||||
Any documents marked `addToPendingDocuments` will appear in the connector cursor; `DownloadCvrf` can remain disabled if the ZIP artefact is pre-seeded.
|
||||
|
||||
## 5. Outstanding items
|
||||
|
||||
- Ops to confirm tenant/app names and provide client credentials through the secure channel.
|
||||
- Connector team monitors token cache health (already implemented); validate instrumentation once Ops supplies credentials.
|
||||
- Offline Kit packaging: add encrypted blob containing client credentials with rotation instructions.
|
||||
# Concelier MSRC Connector - Microsoft Entra Onboarding Brief
|
||||
|
||||
_Last updated: 2026-04-22_
|
||||
|
||||
## 1. App registration requirements
|
||||
|
||||
Sign in at `https://entra.microsoft.com` and open **App registrations**.
|
||||
|
||||
Create or select a confidential client application that will use client credentials. Capture:
|
||||
|
||||
- `tenantId` = Directory (tenant) ID
|
||||
- `clientId` = Application (client) ID
|
||||
- `clientSecret` = client secret value from **Certificates & secrets**
|
||||
|
||||
Primary operator path:
|
||||
|
||||
- Web UI: **Security Posture -> Configure Sources** or **Ops -> Operations -> Feeds & Airgap -> Configure Sources**
|
||||
- CLI:
|
||||
```bash
|
||||
stella db connectors configure microsoft \
|
||||
--server https://concelier.example.internal \
|
||||
--set tenantId=... \
|
||||
--set clientId=... \
|
||||
--set clientSecret=...
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Microsoft recommends certificates over client secrets for production applications. The current StellaOps MSRC operator entry path is client-secret based.
|
||||
- MSRC is not documented here as a separate paid product, but you do need a Microsoft Entra tenant plus permission to register the app and grant the required consent.
|
||||
|
||||
## 2. Secret and credential policy
|
||||
|
||||
- Maintain two client secrets rotating every 90 days.
|
||||
- Prefer storing the active secret through the StellaOps source configuration UI or CLI.
|
||||
- Offline Kit deployments that still mirror secrets out of band must mirror the secret payloads in their encrypted store.
|
||||
- Legacy host configuration remains a compatibility fallback only.
|
||||
|
||||
## 3. Compatibility fallback (`concelier.yaml`)
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
microsoft:
|
||||
tenantId: "<azure-tenant-guid>"
|
||||
clientId: "<app-registration-client-id>"
|
||||
clientSecret: "<pull from secret store>"
|
||||
apiVersion: "2024-08-01"
|
||||
locale: "en-US"
|
||||
requestDelay: "00:00:00.250"
|
||||
failureBackoff: "00:05:00"
|
||||
cursorOverlapMinutes: 10
|
||||
downloadCvrf: false
|
||||
```
|
||||
|
||||
The runtime source ID is `microsoft`, and the connector still binds the legacy `vndr:msrc` section for compatibility.
|
||||
|
||||
## 4. CVRF artefacts
|
||||
|
||||
- The MSRC REST payload exposes `cvrfUrl` per advisory.
|
||||
- Current connector behavior records the link as advisory metadata and reference; it does not download the ZIP by default.
|
||||
- Mirror CVRF ZIPs when preparing Offline Kits if air-gapped deployments need them.
|
||||
|
||||
## 5. Outstanding items
|
||||
|
||||
- Ops must confirm the tenant and app used for production MSRC access.
|
||||
- Connector owners should validate token cache health once credentials are supplied.
|
||||
- If certificate-based auth is later required, StellaOps needs a dedicated certificate entry path before switching the runbook away from client secrets.
|
||||
|
||||
@@ -1,26 +1,75 @@
|
||||
# Concelier Oracle CPU Connector - Operations Runbook
|
||||
|
||||
_Last updated: 2026-01-16_
|
||||
_Last updated: 2026-04-22_
|
||||
|
||||
## 1. Overview
|
||||
The Oracle connector ingests Oracle Critical Patch Update advisories and maps them to canonical IDs.
|
||||
|
||||
The Concelier Oracle connector ingests Oracle Critical Patch Update advisories and maps them to canonical IDs.
|
||||
|
||||
For the default Excititor VEX mirror bootstrap, Stella Ops discovers Oracle updates from Oracle public security RSS and derives the matching CSAF JSON document URIs deterministically.
|
||||
|
||||
## 2. Authentication
|
||||
|
||||
- No authentication required for public advisories.
|
||||
|
||||
## 3. Configuration (`concelier.yaml`)
|
||||
## 3. Configuration paths
|
||||
|
||||
Primary operator path:
|
||||
|
||||
- Web UI: **Security Posture -> Configure Sources** or **Ops -> Operations -> Feeds & Airgap -> Configure Sources**
|
||||
- CLI:
|
||||
```bash
|
||||
stella db connectors configure oracle \
|
||||
--server https://concelier.example.internal \
|
||||
--set calendarUris=https://www.oracle.com/security-alerts/
|
||||
```
|
||||
|
||||
Use the UI/CLI path only when overriding the default Oracle landing page or pinning specific advisories for mirrored or offline flows. Leaving the URI fields blank keeps the built-in public default.
|
||||
|
||||
Compatibility fallback (`concelier.yaml`):
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
oracle:
|
||||
baseUri: "<oracle-cpu-feed-base>"
|
||||
maxDocumentsPerFetch: 20
|
||||
fetchTimeout: "00:00:45"
|
||||
requestDelay: "00:00:00"
|
||||
vndr:
|
||||
oracle:
|
||||
calendarUris:
|
||||
- "https://www.oracle.com/security-alerts/"
|
||||
advisoryUris: []
|
||||
requestDelay: "00:00:01"
|
||||
```
|
||||
|
||||
## 4. Offline and air-gapped deployments
|
||||
- Mirror CPU advisories into the Offline Kit and repoint `baseUri` to the mirror.
|
||||
Notes:
|
||||
|
||||
## 5. Common failure modes
|
||||
- Schedule drift during quarterly CPU updates.
|
||||
- Concelier uses Oracle calendar or index pages for the advisory connector.
|
||||
- `calendarUris` and `advisoryUris` are both valid.
|
||||
- At least one of them must be configured unless you rely on the built-in default Oracle landing page.
|
||||
- `calendarUris` should point at Oracle calendar or index pages; the connector filters discovered links down to concrete `/security-alerts/*.html` advisory pages.
|
||||
|
||||
Example: pinning a specific quarterly CPU advisory directly
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
vndr:
|
||||
oracle:
|
||||
advisoryUris:
|
||||
- "https://www.oracle.com/security-alerts/cpuapr2025.html"
|
||||
requestDelay: "00:00:00"
|
||||
```
|
||||
|
||||
## 4. Excititor default public VEX bootstrap
|
||||
|
||||
- Discovery RSS: `https://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/rss-otn-sec.xml`
|
||||
- Derived CSAF base: `https://www.oracle.com/docs/tech/security-alerts/`
|
||||
|
||||
## 5. Offline and air-gapped deployments
|
||||
|
||||
- Mirror CPU advisories into the Offline Kit and repoint `calendarUris` or `advisoryUris` to the mirrored HTML endpoints for advisory ingestion.
|
||||
- For Excititor mirror bootstrap, mirror the Oracle RSS feed and the derived `/docs/tech/security-alerts/*csaf.json` documents.
|
||||
|
||||
## 6. Common failure modes
|
||||
|
||||
- Schedule drift during quarterly CPU updates
|
||||
- Oracle renaming RSS items or advisory stems in a way that changes the derived CSAF JSON path
|
||||
- Mirroring only the HTML pages while omitting the derived CSAF JSON payloads needed by Excititor
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
# Concelier Ubuntu USN Connector - Operations Runbook
|
||||
|
||||
_Last updated: 2026-01-16_
|
||||
_Last updated: 2026-04-21_
|
||||
|
||||
## 1. Overview
|
||||
The Ubuntu connector ingests Ubuntu Security Notices (USN) and maps advisories to Ubuntu package versions.
|
||||
The Concelier Ubuntu connector ingests Ubuntu Security Notices (USN) and maps advisories to Ubuntu package versions.
|
||||
|
||||
The same public notice feed also backs the default Excititor VEX mirror bootstrap. Ubuntu does not currently publish native CSAF for this path, so Excititor synthesizes deterministic CSAF documents from the notice JSON while preserving the upstream source URI in metadata.
|
||||
|
||||
## 2. Authentication
|
||||
- No authentication required for public feeds.
|
||||
@@ -19,8 +21,23 @@ concelier:
|
||||
requestDelay: "00:00:00"
|
||||
```
|
||||
|
||||
## 4. Offline and air-gapped deployments
|
||||
- Mirror USN feeds into the Offline Kit and repoint `baseUri` to the mirror.
|
||||
## 4. Excititor default public VEX bootstrap
|
||||
- Index URI: `https://ubuntu.com/security/notices.json`
|
||||
- Notice detail base URI: `https://ubuntu.com/security/notices/`
|
||||
- Default page size: `20`
|
||||
- Default max notices per fetch: `60`
|
||||
- Default resume overlap: `3.00:00:00`
|
||||
|
||||
## 5. Common failure modes
|
||||
Operational guidance:
|
||||
- Keep the small page size and bounded fetch count unless Canonical publishes a stronger bulk-ingest contract. This avoids burst-fetching the full notice history during mirror bootstrap.
|
||||
- Keep the resume overlap enabled so the mirror rechecks recently updated notices without needing a full backfill.
|
||||
- Mirror both the paged `notices.json` index responses and the per-notice `USN-xxxx-x.json` documents for offline kits.
|
||||
|
||||
## 5. Offline and air-gapped deployments
|
||||
- Mirror USN feeds into the Offline Kit and repoint `baseUri` to the mirror for advisory ingestion.
|
||||
- For Excititor mirror bootstrap, mirror the `notices.json` index plus the per-notice JSON documents under the same path layout so synthesized CSAF documents remain deterministic.
|
||||
|
||||
## 6. Common failure modes
|
||||
- USN schema updates or missing release references.
|
||||
- Per-notice JSON documents lagging behind the index update window.
|
||||
- Overly aggressive page sizes or fetch counts causing avoidable upstream pressure during first-run bootstrap.
|
||||
|
||||
135
docs/modules/concelier/operations/source-credentials.md
Normal file
135
docs/modules/concelier/operations/source-credentials.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# Advisory Source Credential Entry
|
||||
|
||||
_Last updated: 2026-04-22_
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Stella Ops now supports operator-supplied advisory source settings through the product surfaces that operators already use:
|
||||
|
||||
- Web UI source management
|
||||
- `stella db connectors configure ...` in the CLI
|
||||
|
||||
Environment variables and host-local `concelier.yaml` values remain compatibility fallbacks for older deployments, but the primary operator path for supported advisory sources is now persisted source configuration owned by Concelier itself.
|
||||
|
||||
## 2. Operator entry paths
|
||||
|
||||
### Web UI
|
||||
|
||||
Use either of these routes:
|
||||
|
||||
- **Security Posture -> Configure Sources**
|
||||
- **Ops -> Operations -> Feeds & Airgap -> Configure Sources**
|
||||
|
||||
Then:
|
||||
|
||||
1. Expand the source card.
|
||||
2. Open **Stored Connector Configuration**.
|
||||
3. Enter or update the source fields.
|
||||
4. Save the configuration.
|
||||
|
||||
Sensitive fields never round-trip back to the browser. A stored secret is shown only as retained state. Leaving a password field blank keeps the retained secret. Explicitly checking the clear control removes the stored secret.
|
||||
|
||||
### CLI
|
||||
|
||||
Inspect current persisted source configuration:
|
||||
|
||||
```bash
|
||||
stella db connectors configure ghsa --server https://concelier.example.internal
|
||||
stella db connectors configure cisco --server https://concelier.example.internal
|
||||
```
|
||||
|
||||
Update a source:
|
||||
|
||||
```bash
|
||||
stella db connectors configure ghsa \
|
||||
--server https://concelier.example.internal \
|
||||
--set apiToken=github_pat_xxx
|
||||
|
||||
stella db connectors configure cisco \
|
||||
--server https://concelier.example.internal \
|
||||
--set clientId=... \
|
||||
--set clientSecret=...
|
||||
|
||||
stella db connectors configure microsoft \
|
||||
--server https://concelier.example.internal \
|
||||
--set tenantId=... \
|
||||
--set clientId=... \
|
||||
--set clientSecret=...
|
||||
|
||||
stella db connectors configure oracle \
|
||||
--server https://concelier.example.internal \
|
||||
--set calendarUris=https://www.oracle.com/security-alerts/,https://mirror.example.internal/oracle/
|
||||
|
||||
stella db connectors configure adobe \
|
||||
--server https://concelier.example.internal \
|
||||
--set indexUri=https://mirror.example.internal/adobe/security-bulletin.html \
|
||||
--set additionalIndexUris=https://mirror.example.internal/adobe/archive-1.html;https://mirror.example.internal/adobe/archive-2.html
|
||||
|
||||
stella db connectors configure chromium \
|
||||
--server https://concelier.example.internal \
|
||||
--set feedUri=https://mirror.example.internal/chromium/atom.xml
|
||||
```
|
||||
|
||||
Clear stored fields:
|
||||
|
||||
```bash
|
||||
stella db connectors configure ghsa \
|
||||
--server https://concelier.example.internal \
|
||||
--clear apiToken
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `--set` accepts `key=value`.
|
||||
- Multi-value URI fields such as `calendarUris`, `advisoryUris`, and `additionalIndexUris` accept comma-, semicolon-, or newline-separated absolute URIs.
|
||||
- The current CLI path places literal values on the command line. If shell-history exposure is unacceptable for a secret, prefer the Web UI path or use an operator-approved secure shell/history procedure.
|
||||
|
||||
## 3. Credential acquisition matrix
|
||||
|
||||
| Source | Where to sign in or look | What to create or capture | Can the config be skipped? | Entitlement / paywall notes |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `ghsa` | `https://github.com/settings/personal-access-tokens` or a GitHub App owned by your org | `apiToken` | Not for the current StellaOps GHSA connector path. The upstream API can expose public reviewed advisories anonymously, but the current StellaOps connector still expects a token. | No separate GHSA paywall. GitHub org PAT policy or SAML SSO may require approval or token authorization. |
|
||||
| `cisco` | `https://apiconsole.cisco.com` -> **My Apps & Keys** | `clientId`, `clientSecret` for a Service / Client Credentials app bound to Cisco PSIRT openVuln API | Not for the authenticated Concelier Cisco advisory connector. | No separate StellaOps-side fee, but a Cisco.com account, terms acceptance, and visible openVuln entitlement are required. This is separate from the public Cisco CSAF VEX bootstrap, which stays credential-free. |
|
||||
| `microsoft` | `https://entra.microsoft.com` -> **App registrations** | `tenantId`, `clientId`, `clientSecret` for a confidential client allowed to use MSRC client credentials | Not for the MSRC advisory connector. | No separate documented MSRC paywall, but you need a Microsoft Entra tenant plus permission to register the app and grant the required consent. |
|
||||
| `oracle` | Public Oracle security pages | Usually nothing. Optionally capture mirrored `calendarUris` or pinned `advisoryUris`. | Yes, if the default Oracle security alerts landing page is acceptable. Configure it only when pinning or mirroring. | Public, no login or paywall required for the default path. |
|
||||
| `adobe` | Public Adobe bulletin index | Usually nothing. Optionally capture a mirrored `indexUri` and `additionalIndexUris`. | Yes, for the default public Adobe index. Configure it only when overriding or mirroring the public endpoints. | Public, no login or paywall required for the default path. |
|
||||
| `chromium` | Public Chrome Releases Atom feed | Usually nothing. Optionally capture a mirrored `feedUri`. | Yes, for the default public Chromium feed. Configure it only when overriding or mirroring the public endpoint. | Public, no login or paywall required for the default path. |
|
||||
|
||||
## 4. What operators should actually look for
|
||||
|
||||
### GHSA
|
||||
|
||||
- Personal access token page or org-owned GitHub App credentials
|
||||
- If the organization enforces SAML SSO or PAT approval, make sure the token is authorized for the target org after creation
|
||||
- The StellaOps field name is `apiToken`
|
||||
|
||||
### Cisco
|
||||
|
||||
- Cisco API Console entry for **Cisco PSIRT openVuln API**
|
||||
- Application type: `Service`
|
||||
- Grant type: `Client Credentials`
|
||||
- Capture the generated `clientId` and `clientSecret`
|
||||
|
||||
### Microsoft / MSRC
|
||||
|
||||
- Microsoft Entra **Application (client) ID**
|
||||
- Microsoft Entra **Directory (tenant) ID**
|
||||
- A newly created **Client secret** value
|
||||
- Confirm admin consent and the app permissions expected by your MSRC onboarding process before storing the values in StellaOps
|
||||
|
||||
### Oracle / Adobe / Chromium
|
||||
|
||||
- No credential creation is required
|
||||
- Only collect alternate URIs if you are pointing Concelier at an approved internal mirror or pinning a specific public advisory page
|
||||
|
||||
## 5. References
|
||||
|
||||
- GitHub PAT management: <https://docs.github.com/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token>
|
||||
- GitHub global security advisories REST API: <https://docs.github.com/en/enterprise-cloud@latest/rest/security-advisories/global-advisories>
|
||||
- GitHub SSO authorization for PATs: <https://docs.github.com/articles/authorizing-a-personal-access-token-for-use-with-a-saml-single-sign-on-organization>
|
||||
- Cisco PSIRT openVuln authentication: <https://developer.cisco.com/docs/psirt/authentication/>
|
||||
- Microsoft Entra app registration quickstart: <https://learn.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app>
|
||||
- Microsoft Entra application credentials: <https://learn.microsoft.com/en-us/entra/identity-platform/how-to-add-credentials>
|
||||
- Adobe bulletin index: <https://helpx.adobe.com/security/security-bulletin.html>
|
||||
- Oracle security alerts: <https://www.oracle.com/security-alerts/>
|
||||
- Chrome Releases Atom feed: <https://chromereleases.googleblog.com/atom.xml>
|
||||
Reference in New Issue
Block a user