AGENTS

Role

Shared connector toolkit. Provides HTTP clients, retry/backoff, conditional GET (ETag/Last-Modified), schema validation, pagination helpers, clocks, and common DTO utilities for all connectors.

Scope

Typed HttpClient registrations with allowlisted hosts and timeouts.
Request pipeline: retries with jitter, backoff on 429/5xx, rate-limit tracking per source.
Conditional GET helpers (If-None-Match, If-Modified-Since), window cursors, and pagination iterators.
Validators: JSON Schema, XML Schema (for example XmlSchemaValidator), and sanitizers.
Content hashing and raw document capture helpers; metadata extraction (headers, status).
HTML sanitization, URL normalization, and PDF-to-text extraction utilities for feeds that require cleanup before validation.

Participants

Source.* connectors (NVD, Red Hat, JVN, PSIRTs, CERTs, ICS).
Storage.Mongo (document/dto repositories using shared shapes).
Core (jobs schedule/trigger for connectors).
QA (canned HTTP server harness, schema fixtures).

Interfaces & contracts

All network calls must pass through configured HttpClient with allowlist and sane timeouts; no direct new HttpClient().
Validators return detailed errors; invalid payloads quarantined and not mapped.
Cursor helpers implement sliding windows and ID-based pagination; rely on IClock/TimeProvider for determinism.
Strict provenance tags for extraction method: parser, oval, package.nevra, llm (gated).

In/Out of scope

In: HTTP plumbing, validators, cursor/backoff utilities, hashing. Out: connector-specific schemas/mapping rules, merge precedence.

Observability & security expectations

Metrics: SourceDiagnostics publishes concelier.source.http.* counters/histograms tagged with concelier.source=<connector> plus retries/failures; connector dashboards slice on that tag instead of bespoke metric names.
Logs include uri, status, retries, etag; redact tokens and auth headers.
Distributed tracing hooks and per-connector counters should be wired centrally for consistent observability.

Tests

Author and review coverage in ../StellaOps.Concelier.Connector.Common.Tests.
Shared fixtures (e.g., MongoIntegrationFixture, ConnectorTestHarness) live in ../StellaOps.Concelier.Testing.
Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.

2.3 KiB Raw Blame History