# AGENTS
## Role
Shared connector toolkit. Provides HTTP clients, retry/backoff, conditional GET (ETag/Last-Modified), schema validation, pagination helpers, clocks, and common DTO utilities for all connectors.
## Scope
- Typed HttpClient registrations with allowlisted hosts and timeouts.
- Request pipeline: retries with jitter, backoff on 429/5xx, rate-limit tracking per source.
- Conditional GET helpers (If-None-Match, If-Modified-Since), window cursors, and pagination iterators.
- Validators: JSON Schema, XML Schema (for example XmlSchemaValidator), and sanitizers.
- Content hashing and raw document capture helpers; metadata extraction (headers, status).
- HTML sanitization, URL normalization, and PDF-to-text extraction utilities for feeds that require cleanup before validation.
## Participants
- Source.* connectors (NVD, Red Hat, JVN, PSIRTs, CERTs, ICS).
- Storage.Mongo (document/dto repositories using shared shapes).
- Core (jobs schedule/trigger for connectors).
- QA (canned HTTP server harness, schema fixtures).
## Interfaces & contracts
- All network calls must pass through configured HttpClient with allowlist and sane timeouts; no direct new HttpClient().
- Validators return detailed errors; invalid payloads quarantined and not mapped.
- Cursor helpers implement sliding windows and ID-based pagination; rely on IClock/TimeProvider for determinism.
- Strict provenance tags for extraction method: parser, oval, package.nevra, llm (gated).
## In/Out of scope
In: HTTP plumbing, validators, cursor/backoff utilities, hashing.
Out: connector-specific schemas/mapping rules, merge precedence.
## Observability & security expectations
- Metrics: SourceDiagnostics publishes `concelier.source.http.*` counters/histograms tagged with `concelier.source=<connector>` plus retries/failures; connector dashboards slice on that tag instead of bespoke metric names.
- Logs include uri, status, retries, etag; redact tokens and auth headers.
- Distributed tracing hooks and per-connector counters should be wired centrally for consistent observability.
## Tests
- Author and review coverage in `../StellaOps.Concelier.Connector.Common.Tests`.
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.