master
607e72e2a1
Some checks failed
Build Test Deploy / docs (push) Has been cancelled
Build Test Deploy / deploy (push) Has been cancelled
Build Test Deploy / build-test (push) Has been cancelled
Build Test Deploy / authority-container (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
78 lines
5.3 KiB
Markdown
78 lines
5.3 KiB
Markdown
# Feedser Apple Security Update Connector Operations
|
||
|
||
This runbook covers staging and production rollout for the Apple security updates connector (`source:vndr-apple:*`), including observability checks and fixture maintenance.
|
||
|
||
## 1. Prerequisites
|
||
|
||
- Network egress (or mirrored cache) for `https://gdmf.apple.com/v2/pmv` and the Apple Support domain (`https://support.apple.com/`).
|
||
- Optional: corporate proxy exclusions for the Apple hosts if outbound traffic is normally filtered.
|
||
- Updated configuration (environment variables or `feedser.yaml`) with an `apple` section. Example baseline:
|
||
|
||
```yaml
|
||
feedser:
|
||
sources:
|
||
apple:
|
||
softwareLookupUri: "https://gdmf.apple.com/v2/pmv"
|
||
advisoryBaseUri: "https://support.apple.com/"
|
||
localeSegment: "en-us"
|
||
maxAdvisoriesPerFetch: 25
|
||
initialBackfill: "120.00:00:00"
|
||
modifiedTolerance: "02:00:00"
|
||
failureBackoff: "00:05:00"
|
||
```
|
||
|
||
> ℹ️ `softwareLookupUri` and `advisoryBaseUri` must stay absolute and aligned with the HTTP allow-list; Feedser automatically adds both hosts to the connector HttpClient.
|
||
|
||
## 2. Staging Smoke Test
|
||
|
||
1. Deploy the configuration and restart the Feedser workers to ensure the Apple connector options are bound.
|
||
2. Trigger a full connector cycle:
|
||
- CLI: `stella db jobs run source:vndr-apple:fetch --and-then source:vndr-apple:parse --and-then source:vndr-apple:map`
|
||
- REST: `POST /jobs/run { "kind": "source:vndr-apple:fetch", "chain": ["source:vndr-apple:parse", "source:vndr-apple:map"] }`
|
||
3. Validate metrics exported under meter `StellaOps.Feedser.Source.Vndr.Apple`:
|
||
- `apple.fetch.items` (documents fetched)
|
||
- `apple.fetch.failures`
|
||
- `apple.fetch.unchanged`
|
||
- `apple.parse.failures`
|
||
- `apple.map.affected.count` (histogram of affected package counts)
|
||
4. Cross-check the shared HTTP counters:
|
||
- `feedser.source.http.requests_total{feedser_source="vndr-apple"}` should increase for both index and detail phases.
|
||
- `feedser.source.http.failures_total{feedser_source="vndr-apple"}` should remain flat (0) during a healthy run.
|
||
5. Inspect the info logs:
|
||
- `Apple software index fetch … processed=X newDocuments=Y`
|
||
- `Apple advisory parse complete … aliases=… affected=…`
|
||
- `Mapped Apple advisory … pendingMappings=0`
|
||
6. Confirm MongoDB state:
|
||
- `raw_documents` store contains the HT article HTML with metadata (`apple.articleId`, `apple.postingDate`).
|
||
- `dtos` store has `schemaVersion="apple.security.update.v1"`.
|
||
- `advisories` collection includes keys `HTxxxxxx` with normalized SemVer rules.
|
||
- `source_states` entry for `apple` shows a recent `cursor.lastPosted`.
|
||
|
||
## 3. Production Monitoring
|
||
|
||
- **Dashboards** – Add the following expressions to your Feedser Grafana board (OTLP/Prometheus naming assumed):
|
||
- `rate(apple_fetch_items_total[15m])` vs `rate(feedser_source_http_requests_total{feedser_source="vndr-apple"}[15m])`
|
||
- `rate(apple_fetch_failures_total[5m])` for error spikes (`severity=warning` at `>0`)
|
||
- `histogram_quantile(0.95, rate(apple_map_affected_count_bucket[1h]))` to watch affected-package fan-out
|
||
- `increase(apple_parse_failures_total[6h])` to catch parser drift (alerts at `>0`)
|
||
- **Alerts** – Page if `rate(apple_fetch_items_total[2h]) == 0` during business hours while other connectors are active. This often indicates lookup feed failures or misconfigured allow-lists.
|
||
- **Logs** – Surface warnings `Apple document {DocumentId} missing GridFS payload` or `Apple parse failed`—repeated hits imply storage issues or HTML regressions.
|
||
- **Telemetry pipeline** – `StellaOps.Feedser.WebService` now exports `StellaOps.Feedser.Source.Vndr.Apple` alongside existing Feedser meters; ensure your OTEL collector or Prometheus scraper includes it.
|
||
|
||
## 4. Fixture Maintenance
|
||
|
||
Regression fixtures live under `src/StellaOps.Feedser.Source.Vndr.Apple.Tests/Apple/Fixtures`. Refresh them whenever Apple reshapes the HT layout or when new platforms appear.
|
||
|
||
1. Run the helper script matching your platform:
|
||
- Bash: `./scripts/update-apple-fixtures.sh`
|
||
- PowerShell: `./scripts/update-apple-fixtures.ps1`
|
||
2. Each script exports `UPDATE_APPLE_FIXTURES=1`, updates the `WSLENV` passthrough, and touches `.update-apple-fixtures` so WSL+VS Code test runs observe the flag. The subsequent test execution fetches the live HT articles listed in `AppleFixtureManager`, sanitises the HTML, and rewrites the `.expected.json` DTO snapshots.
|
||
3. Review the diff for localisation or nav noise. Once satisfied, re-run the tests without the env var (`dotnet test src/StellaOps.Feedser.Source.Vndr.Apple.Tests/StellaOps.Feedser.Source.Vndr.Apple.Tests.csproj`) to verify determinism.
|
||
4. Commit fixture updates together with any parser/mapping changes that motivated them.
|
||
|
||
## 5. Known Issues & Follow-up Tasks
|
||
|
||
- Apple occasionally throttles anonymous requests after bursts. The connector backs off automatically, but persistent `apple.fetch.failures` spikes might require mirroring the HT content or scheduling wider fetch windows.
|
||
- Rapid Security Responses may appear before the general patch notes surface in the lookup JSON. When that happens, the fetch run will log `detailFailures>0`. Collect sample HTML and refresh fixtures to confirm parser coverage.
|
||
- Multi-locale content is still under regression sweep (`src/StellaOps.Feedser.Source.Vndr.Apple/TASKS.md`). Capture non-`en-us` snapshots once the fixture tooling stabilises.
|