32 lines
2.3 KiB
Markdown
32 lines
2.3 KiB
Markdown
# AGENTS
|
|
## Role
|
|
Shared connector toolkit. Provides HTTP clients, retry/backoff, conditional GET (ETag/Last-Modified), schema validation, pagination helpers, clocks, and common DTO utilities for all connectors.
|
|
## Scope
|
|
- Typed HttpClient registrations with allowlisted hosts and timeouts.
|
|
- Request pipeline: retries with jitter, backoff on 429/5xx, rate-limit tracking per source.
|
|
- Conditional GET helpers (If-None-Match, If-Modified-Since), window cursors, and pagination iterators.
|
|
- Validators: JSON Schema, XML Schema (for example XmlSchemaValidator), and sanitizers.
|
|
- Content hashing and raw document capture helpers; metadata extraction (headers, status).
|
|
- HTML sanitization, URL normalization, and PDF-to-text extraction utilities for feeds that require cleanup before validation.
|
|
## Participants
|
|
- Source.* connectors (NVD, Red Hat, JVN, PSIRTs, CERTs, ICS).
|
|
- Storage.Mongo (document/dto repositories using shared shapes).
|
|
- Core (jobs schedule/trigger for connectors).
|
|
- QA (canned HTTP server harness, schema fixtures).
|
|
## Interfaces & contracts
|
|
- All network calls must pass through configured HttpClient with allowlist and sane timeouts; no direct new HttpClient().
|
|
- Validators return detailed errors; invalid payloads quarantined and not mapped.
|
|
- Cursor helpers implement sliding windows and ID-based pagination; rely on IClock/TimeProvider for determinism.
|
|
- Strict provenance tags for extraction method: parser, oval, package.nevra, llm (gated).
|
|
## In/Out of scope
|
|
In: HTTP plumbing, validators, cursor/backoff utilities, hashing.
|
|
Out: connector-specific schemas/mapping rules, merge precedence.
|
|
## Observability & security expectations
|
|
- Metrics: SourceDiagnostics publishes `concelier.source.http.*` counters/histograms tagged with `concelier.source=<connector>` plus retries/failures; connector dashboards slice on that tag instead of bespoke metric names.
|
|
- Logs include uri, status, retries, etag; redact tokens and auth headers.
|
|
- Distributed tracing hooks and per-connector counters should be wired centrally for consistent observability.
|
|
## Tests
|
|
- Author and review coverage in `../StellaOps.Concelier.Connector.Common.Tests`.
|
|
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
|
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|