Add unit and integration tests for VexCandidateEmitter and SmartDiff repositories
- Implemented comprehensive unit tests for VexCandidateEmitter to validate candidate emission logic based on various scenarios including absent and present APIs, confidence thresholds, and rate limiting. - Added integration tests for SmartDiff PostgreSQL repositories, covering snapshot storage and retrieval, candidate storage, and material risk change handling. - Ensured tests validate correct behavior for storing, retrieving, and querying snapshots and candidates, including edge cases and expected outcomes.
This commit is contained in:
master
@@ -1,13 +1,13 @@
|
||||
# Concelier · AGENTS Charter (Sprint 0112–0114)
|
||||
|
||||
## Module Scope & Working Directory
|
||||
- Working directory: `src/Concelier/**` (WebService, __Libraries, Storage.Mongo, analyzers, tests, seed-data). Do not edit other modules unless explicitly referenced by this sprint.
|
||||
- Working directory: `src/Concelier/**` (WebService, __Libraries, Storage.Postgres, analyzers, tests, seed-data). Do not edit other modules unless explicitly referenced by this sprint.
|
||||
- Mission: Link-Not-Merge (LNM) ingestion of advisory observations, correlation into linksets, evidence/export APIs, and deterministic telemetry.
|
||||
|
||||
## Roles
|
||||
- **Backend engineer (ASP.NET Core / Mongo):** connectors, ingestion guards, linkset builder, WebService APIs, storage migrations.
|
||||
- **Backend engineer (ASP.NET Core / PostgreSQL):** connectors, ingestion guards, linkset builder, WebService APIs, storage migrations.
|
||||
- **Observability/Platform engineer:** OTEL metrics/logs, health/readiness, distributed locks, scheduler safety.
|
||||
- **QA automation:** Mongo2Go + WebApplicationFactory tests for handlers/jobs; determinism and guardrail regression harnesses.
|
||||
- **QA automation:** Testcontainers + WebApplicationFactory tests for handlers/jobs; determinism and guardrail regression harnesses.
|
||||
- **Docs/Schema steward:** keep LNM schemas, API references, and inline provenance docs aligned with behavior.
|
||||
|
||||
## Required Reading (must be treated as read before setting DOING)
|
||||
@@ -34,16 +34,16 @@
|
||||
|
||||
## Coding & Observability Standards
|
||||
- Target **.NET 10**; prefer latest C# preview features already enabled in repo.
|
||||
- Mongo driver ≥ 3.x; canonical BSON/JSON mapping lives in Storage.Mongo.
|
||||
- Npgsql driver for PostgreSQL; canonical JSON mapping in Storage.Postgres.
|
||||
- Metrics: use `Meter` names under `StellaOps.Concelier.*`; tag `tenant`, `source`, `result` as applicable. Counters/histograms must be documented.
|
||||
- Logging: structured, no PII; include `tenant`, `source`, `job`, `correlationId` when available.
|
||||
- Scheduler/locks: one lock per connector/export job; no duplicate runs; honor `CancellationToken`.
|
||||
|
||||
## Testing Rules
|
||||
- Write/maintain tests alongside code:
|
||||
- Web/API: `StellaOps.Concelier.WebService.Tests` with WebApplicationFactory + Mongo2Go fixtures.
|
||||
- Web/API: `StellaOps.Concelier.WebService.Tests` with WebApplicationFactory + Testcontainers fixtures.
|
||||
- Core/Linkset/Guards: `StellaOps.Concelier.Core.Tests`.
|
||||
- Storage: `StellaOps.Concelier.Storage.Mongo.Tests` (use in-memory or Mongo2Go; determinism on ordering/hashes).
|
||||
- Storage: `StellaOps.Concelier.Storage.Postgres.Tests` (use in-memory or Testcontainers; determinism on ordering/hashes).
|
||||
- Observability/analyzers: tests in `__Analyzers` or respective test projects.
|
||||
- Tests must assert determinism (stable ordering/hashes), tenant guards, AOC invariants, and no derived fields in ingestion.
|
||||
- Prefer seeded fixtures under `seed-data/` for repeatability; avoid network in tests.
|
||||
|
||||
@@ -11,13 +11,13 @@ Bootstrap the ACSC (Australian Cyber Security Centre) advisories connector so th
|
||||
|
||||
## Participants
|
||||
- `Source.Common` for HTTP client creation, fetch service, and DTO persistence helpers.
|
||||
- `Storage.Mongo` for raw/document/DTO/advisory storage plus cursor management.
|
||||
- `Storage.Postgres` for raw/document/DTO/advisory storage plus cursor management.
|
||||
- `Concelier.Models` for canonical advisory structures and provenance utilities.
|
||||
- `Concelier.Testing` for integration harnesses and snapshot helpers.
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds should follow the pattern `acsc:fetch`, `acsc:parse`, `acsc:map`.
|
||||
- Documents persisted to Mongo must include ETag/Last-Modified metadata when the source exposes it.
|
||||
- Documents persisted to PostgreSQL must include ETag/Last-Modified metadata when the source exposes it.
|
||||
- Canonical advisories must emit aliases (ACSC ID + CVE IDs) and references (official bulletin + vendor notices).
|
||||
|
||||
## In/Out of scope
|
||||
|
||||
@@ -11,7 +11,7 @@ Build the CCCS (Canadian Centre for Cyber Security) advisories connector so Conc
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP clients, fetch service, DTO storage helpers).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores + source state).
|
||||
- `Storage.Postgres` (raw/document/DTO/advisory stores + source state).
|
||||
- `Concelier.Models` (canonical advisory data structures).
|
||||
- `Concelier.Testing` (integration fixtures and snapshot utilities).
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ Deliver a connector for Germany’s CERT-Bund advisories so Concelier can ingest
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Storage.Postgres` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical data model).
|
||||
- `Concelier.Testing` (integration harness, snapshot utilities).
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ Implement the CERT/CC (Carnegie Mellon CERT Coordination Center) advisory connec
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores and state).
|
||||
- `Storage.Postgres` (raw/document/DTO/advisory stores and state).
|
||||
- `Concelier.Models` (canonical structures).
|
||||
- `Concelier.Testing` (integration tests and snapshots).
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ ANSSI CERT-FR advisories connector (avis/alertes) providing national enrichment:
|
||||
- Maintain watermarks and de-duplication by content hash; idempotent processing.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML parsing helpers, validators).
|
||||
- Storage.Mongo (document, dto, advisory, reference, source_state).
|
||||
- Storage.Postgres (document, dto, advisory, reference, source_state).
|
||||
- Models (canonical).
|
||||
- Core/WebService (jobs: source:certfr:fetch|parse|map).
|
||||
- Merge engine (later) to enrich only.
|
||||
@@ -23,7 +23,7 @@ Out: OVAL or package-level authority.
|
||||
- Logs: feed URL(s), item ids/urls, extraction durations; no PII; allowlist hostnames.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.CertFr.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -7,7 +7,7 @@ CERT-In national CERT connector; enrichment advisories for India; maps CVE lists
|
||||
- Persist raw docs and maintain source_state cursor; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML parsing, normalization, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, reference, source_state).
|
||||
- Storage.Postgres (document, dto, advisory, alias, reference, source_state).
|
||||
- Models (canonical).
|
||||
- Core/WebService (jobs: source:certin:fetch|parse|map).
|
||||
- Merge engine treats CERT-In as enrichment (no override of PSIRT or OVAL without concrete ranges).
|
||||
@@ -24,7 +24,7 @@ Out: package range authority; scraping behind auth walls.
|
||||
- Logs: advisory codes, CVE counts per advisory, timing; allowlist host; redact personal data if present.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.CertIn.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -10,7 +10,7 @@ Shared connector toolkit. Provides HTTP clients, retry/backoff, conditional GET
|
||||
- HTML sanitization, URL normalization, and PDF-to-text extraction utilities for feeds that require cleanup before validation.
|
||||
## Participants
|
||||
- Source.* connectors (NVD, Red Hat, JVN, PSIRTs, CERTs, ICS).
|
||||
- Storage.Mongo (document/dto repositories using shared shapes).
|
||||
- Storage.Postgres (document/dto repositories using shared shapes).
|
||||
- Core (jobs schedule/trigger for connectors).
|
||||
- QA (canned HTTP server harness, schema fixtures).
|
||||
## Interfaces & contracts
|
||||
@@ -27,7 +27,7 @@ Out: connector-specific schemas/mapping rules, merge precedence.
|
||||
- Distributed tracing hooks and per-connector counters should be wired centrally for consistent observability.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Common.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -11,7 +11,7 @@ Create a dedicated CVE connector when we need raw CVE stream ingestion outside o
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores & source state).
|
||||
- `Storage.Postgres` (raw/document/DTO/advisory stores & source state).
|
||||
- `Concelier.Models` (canonical data model).
|
||||
- `Concelier.Testing` (integration fixtures, snapshot helpers).
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ Red Hat distro connector (Security Data API and OVAL) providing authoritative OS
|
||||
- Map to canonical advisories with affected Type=rpm/cpe, fixedBy NEVRA, RHSA aliasing; persist provenance indicating oval/package.nevra.
|
||||
## Participants
|
||||
- Source.Common (HTTP, throttling, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, source_state).
|
||||
- Storage.Postgres (document, dto, advisory, alias, affected, reference, source_state).
|
||||
- Models (canonical Affected with NEVRA).
|
||||
- Core/WebService (jobs: source:redhat:fetch|parse|map) already registered.
|
||||
- Merge engine to enforce distro precedence (OVAL or PSIRT greater than NVD).
|
||||
@@ -23,7 +23,7 @@ Out: building RPM artifacts; cross-distro reconciliation beyond Red Hat.
|
||||
- Logs: cursor bounds, advisory ids, NEVRA counts; allowlist Red Hat endpoints.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Distro.RedHat.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -11,7 +11,7 @@ Implement a connector for GitHub Security Advisories (GHSA) when we need to inge
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP clients, fetch service, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores and source state).
|
||||
- `Storage.Postgres` (raw/document/DTO/advisory stores and source state).
|
||||
- `Concelier.Models` (canonical advisory types).
|
||||
- `Concelier.Testing` (integration harness, snapshot helpers).
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ Implement the CISA ICS advisory connector to ingest US CISA Industrial Control S
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores + source state).
|
||||
- `Storage.Postgres` (raw/document/DTO/advisory stores + source state).
|
||||
- `Concelier.Models` (canonical advisory structures).
|
||||
- `Concelier.Testing` (integration fixtures and snapshots).
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ Kaspersky ICS-CERT connector; authoritative for OT/ICS vendor advisories covered
|
||||
- Persist raw docs with sha256; maintain source_state; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML helpers, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, source_state).
|
||||
- Storage.Postgres (document, dto, advisory, alias, affected, reference, source_state).
|
||||
- Models (canonical; affected.platform="ics-vendor", tags for device families).
|
||||
- Core/WebService (jobs: source:ics-kaspersky:fetch|parse|map).
|
||||
- Merge engine respects ICS vendor authority for OT impact.
|
||||
@@ -24,7 +24,7 @@ Out: firmware downloads; reverse-engineering artifacts.
|
||||
- Logs: slugs, vendor/product counts, timing; allowlist host.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Ics.Kaspersky.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -7,7 +7,7 @@ Japan JVN/MyJVN connector; national CERT enrichment with strong identifiers (JVN
|
||||
- Persist raw docs with sha256 and headers; manage source_state cursor; idempotent parse/map.
|
||||
## Participants
|
||||
- Source.Common (HTTP, pagination, XML or XSD validators, retries/backoff).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected (when concrete), reference, jp_flags, source_state).
|
||||
- Storage.Postgres (document, dto, advisory, alias, affected (when concrete), reference, jp_flags, source_state).
|
||||
- Models (canonical Advisory/Affected/Provenance).
|
||||
- Core/WebService (jobs: source:jvn:fetch|parse|map).
|
||||
- Merge engine applies enrichment precedence (does not override distro or PSIRT ranges unless JVN gives explicit package truth).
|
||||
@@ -25,7 +25,7 @@ Out: overriding distro or PSIRT ranges without concrete evidence; scraping unoff
|
||||
- Logs: window bounds, jvndb ids processed, vendor_status distribution; redact API keys.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Jvn.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -11,7 +11,7 @@ Implement the CISA Known Exploited Vulnerabilities (KEV) catalogue connector to
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP client, fetch service, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Storage.Postgres` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (advisory + range primitive types).
|
||||
- `Concelier.Testing` (integration fixtures & snapshots).
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ Deliver the KISA (Korea Internet & Security Agency) advisory connector to ingest
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Storage.Postgres` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical data structures).
|
||||
- `Concelier.Testing` (integration fixtures and snapshots).
|
||||
|
||||
|
||||
@@ -22,7 +22,7 @@ Out: authoritative distro package ranges; vendor patch states.
|
||||
- Metrics: SourceDiagnostics publishes `concelier.source.http.*` counters/histograms tagged `concelier.source=nvd`; dashboards slice on the tag to track page counts, schema failures, map throughput, and window advancement. Structured logs include window bounds and etag hits.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Nvd.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -8,7 +8,7 @@ Connector for OSV.dev across ecosystems; authoritative SemVer/PURL ranges for OS
|
||||
- Maintain per-ecosystem cursors and deduplicate runs via payload hashes to keep reruns idempotent.
|
||||
## Participants
|
||||
- Source.Common supplies HTTP clients, pagination helpers, and validators.
|
||||
- Storage.Mongo persists documents, DTOs, advisories, and source_state cursors.
|
||||
- Storage.Postgres persists documents, DTOs, advisories, and source_state cursors.
|
||||
- Merge engine resolves OSV vs GHSA consistency; prefers SemVer data for libraries; distro OVAL still overrides OS packages.
|
||||
- Exporters serialize per-ecosystem ranges untouched.
|
||||
## Interfaces & contracts
|
||||
@@ -22,7 +22,7 @@ Out: vendor PSIRT and distro OVAL specifics.
|
||||
- Metrics: SourceDiagnostics exposes the shared `concelier.source.http.*` counters/histograms tagged `concelier.source=osv`; observability dashboards slice on the tag to monitor item volume, schema failures, range counts, and ecosystem coverage. Logs include ecosystem and cursor values.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Osv.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -11,7 +11,7 @@ Implement the Russian BDU (Vulnerability Database) connector to ingest advisorie
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores + source state).
|
||||
- `Storage.Postgres` (raw/document/DTO/advisory stores + source state).
|
||||
- `Concelier.Models` (canonical data structures).
|
||||
- `Concelier.Testing` (integration harness, snapshot utilities).
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ Implement the Russian NKTsKI (formerly NKCKI) advisories connector to ingest NKT
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Storage.Postgres` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical data structures).
|
||||
- `Concelier.Testing` (integration fixtures, snapshots).
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ Adobe PSIRT connector ingesting APSB/APA advisories; authoritative for Adobe pro
|
||||
- Persist raw docs with sha256 and headers; maintain source_state cursors; ensure idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML parsing, retries/backoff, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Storage.Postgres (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical Advisory/Affected/Provenance).
|
||||
- Core/WebService (jobs: source:adobe:fetch|parse|map).
|
||||
- Merge engine (later) to apply PSIRT override policy for Adobe packages.
|
||||
@@ -24,7 +24,7 @@ Out: signing, package artifact downloads, non-Adobe product truth.
|
||||
- Logs: advisory ids, product counts, extraction timings; hosts allowlisted; no secret logging.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Adobe.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -11,7 +11,7 @@ Implement the Apple security advisories connector to ingest Apple HT/HT2 securit
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Storage.Postgres` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical structures + range primitives).
|
||||
- `Concelier.Testing` (integration fixtures/snapshots).
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ Chromium/Chrome vendor feed connector parsing Stable Channel Update posts; autho
|
||||
- Persist raw docs and maintain source_state cursor; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML helpers, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Storage.Postgres (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical; affected ranges by product/version).
|
||||
- Core/WebService (jobs: source:chromium:fetch|parse|map).
|
||||
- Merge engine (later) to respect vendor PSIRT precedence for Chrome.
|
||||
@@ -24,7 +24,7 @@ Out: OS distro packaging semantics; bug bounty details beyond references.
|
||||
- Logs: post slugs, version extracted, platform coverage, timing; allowlist blog host.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Chromium.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -10,7 +10,7 @@ Implement the Cisco security advisory connector to ingest Cisco PSIRT bulletins
|
||||
- Provide deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common`, `Storage.Mongo`, `Concelier.Models`, `Concelier.Testing`.
|
||||
- `Source.Common`, `Storage.Postgres`, `Concelier.Models`, `Concelier.Testing`.
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `cisco:fetch`, `cisco:parse`, `cisco:map`.
|
||||
|
||||
@@ -10,7 +10,7 @@ Implement the Microsoft Security Response Center (MSRC) connector to ingest Micr
|
||||
- Provide deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common`, `Storage.Mongo`, `Concelier.Models`, `Concelier.Testing`.
|
||||
- `Source.Common`, `Storage.Postgres`, `Concelier.Models`, `Concelier.Testing`.
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `msrc:fetch`, `msrc:parse`, `msrc:map`.
|
||||
|
||||
@@ -7,7 +7,7 @@ Oracle PSIRT connector for Critical Patch Updates (CPU) and Security Alerts; aut
|
||||
- Persist raw documents; maintain source_state across cycles; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Storage.Postgres (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical; affected ranges for vendor products).
|
||||
- Core/WebService (jobs: source:oracle:fetch|parse|map).
|
||||
- Merge engine (later) to prefer PSIRT ranges over NVD for Oracle products.
|
||||
@@ -23,7 +23,7 @@ Out: signing or patch artifact downloads.
|
||||
- Logs: cycle tags, advisory ids, extraction timings; redact nothing sensitive.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Oracle.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -7,7 +7,7 @@ VMware/Broadcom PSIRT connector ingesting VMSA advisories; authoritative for VMw
|
||||
- Persist raw docs with sha256; manage source_state; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, cookies/session handling if needed, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Storage.Postgres (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical).
|
||||
- Core/WebService (jobs: source:vmware:fetch|parse|map).
|
||||
- Merge engine (later) to prefer PSIRT ranges for VMware products.
|
||||
@@ -24,7 +24,7 @@ Out: customer portal authentication flows beyond public advisories; downloading
|
||||
- Logs: vmsa ids, product counts, extraction timings; handle portal rate limits politely.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Vmware.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -10,7 +10,7 @@ Job orchestration and lifecycle. Registers job definitions, schedules execution,
|
||||
- Surfacing: enumerate definitions, last run, recent runs, active runs to WebService endpoints.
|
||||
## Participants
|
||||
- WebService exposes REST endpoints for definitions, runs, active, and trigger.
|
||||
- Storage.Mongo persists job definitions metadata, run documents, and leases (locks collection).
|
||||
- Storage.Postgres persists job definitions metadata, run documents, and leases (locks table).
|
||||
- Source connectors and Exporters implement IJob and are registered into the scheduler via DI and Plugin routines.
|
||||
- Models/Merge/Export are invoked indirectly through jobs.
|
||||
- Plugin host runtime loads dependency injection routines that register job definitions.
|
||||
@@ -27,7 +27,7 @@ Out: business logic of connectors/exporters, HTTP handlers (owned by WebService)
|
||||
- Honor CancellationToken early and often.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Core.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ Optional exporter producing vuln-list-shaped JSON tree for downstream trivy-db b
|
||||
- Packaging: output directory under exports/json/<timestamp> with reproducible naming; optionally symlink latest.
|
||||
- Optional auxiliary index files (for example severity summaries) may be generated when explicitly requested, but must remain deterministic and avoid altering canonical payloads.
|
||||
## Participants
|
||||
- Storage.Mongo.AdvisoryStore as input; ExportState repository for cursors/digests.
|
||||
- Storage.Postgres.AdvisoryStore as input; ExportState repository for cursors/digests.
|
||||
- Core scheduler runs JsonExportJob; Plugin DI wires JsonExporter + job.
|
||||
- TrivyDb exporter may consume the rendered tree in v0 (builder path) if configured.
|
||||
## Interfaces & contracts
|
||||
@@ -23,7 +23,7 @@ Out: ORAS push and Trivy DB BoltDB writing (owned by Trivy exporter).
|
||||
- Logs: target path, record counts, digest; no sensitive data.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Exporter.Json.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ Exporter producing a Trivy-compatible database artifact for self-hosting or offl
|
||||
- DI: TrivyExporter + Jobs.TrivyExportJob registered by TrivyExporterDependencyInjectionRoutine.
|
||||
- Export_state recording: capture digests, counts, start/end timestamps for idempotent reruns and incremental packaging.
|
||||
## Participants
|
||||
- Storage.Mongo.AdvisoryStore as input.
|
||||
- Storage.Postgres.AdvisoryStore as input.
|
||||
- Core scheduler runs export job; WebService/Plugins trigger it.
|
||||
- JSON exporter (optional precursor) if choosing the builder path.
|
||||
## Interfaces & contracts
|
||||
@@ -24,7 +24,7 @@ Out: signing (external pipeline), scanner behavior.
|
||||
- Logs: export path, repo/tag, digest; redact credentials; backoff on push errors.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Exporter.TrivyDb.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ Deterministic merge and reconciliation engine; builds identity graph via aliases
|
||||
- Merge algorithm: stable ordering, pure functions, idempotence; compute beforeHash/afterHash over canonical form; write merge_event.
|
||||
- Conflict reporting: counters and logs for identity conflicts, reference merges, range overrides.
|
||||
## Participants
|
||||
- Storage.Mongo (reads raw mapped advisories, writes merged docs plus merge_event).
|
||||
- Storage.Postgres (reads raw mapped advisories, writes merged docs plus merge_event).
|
||||
- Models (canonical types).
|
||||
- Exporters (consume merged canonical).
|
||||
- Core/WebService (jobs: merge:run, maybe per-kind).
|
||||
@@ -29,7 +29,7 @@ Out: fetching/parsing, exporter packaging, signing.
|
||||
- Logs: decisions (why replaced), keys involved, hashes; avoid dumping large blobs; redact secrets (none expected).
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Merge.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
|
||||
@@ -25,7 +25,7 @@ Out: fetching/parsing external schemas, storage, HTTP.
|
||||
- Emit model version identifiers in logs when canonical structures change; keep adapters for older readers until deprecated.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Models.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Shared fixtures (e.g., `PostgresIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user