feat: Add guild charters and task boards for various components
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Introduced guild charters for Scanner Deno, PHP, Ruby, Native, WebService, Java, Surface.Env, Surface.FS, Surface.Secrets, Surface.Validation, UI, Zastava Observer, Zastava Webhook, Zastava Core, and Plugin Platform. - Each charter outlines the mission, scope, required reading, and working agreements for the respective guilds. - Created task boards for Surface.Env, Surface.FS, Surface.Secrets, Surface.Validation, and Zastava components to track progress and dependencies. - Ensured all documents emphasize determinism, offline readiness, security, and integration with shared Surface libraries.
This commit is contained in:
@@ -1,34 +1,45 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Minimal API host wiring configuration, storage, plugin routines, and job endpoints. Operational surface for health, readiness, and job control.
|
||||
## Scope
|
||||
- Configuration: appsettings.json + etc/concelier.yaml (yaml path = ../etc/concelier.yaml); bind into ConcelierOptions with validation (Only Mongo supported).
|
||||
- Mongo: MongoUrl from options.Storage.Dsn; IMongoClient/IMongoDatabase singletons; default database name fallback (options -> URL -> "concelier").
|
||||
- Services: AddMongoStorage(); AddSourceHttpClients(); RegisterPluginRoutines(configuration, PluginHostOptions).
|
||||
- Bootstrap: MongoBootstrapper.InitializeAsync on startup.
|
||||
- Endpoints (configuration & job control only; root path intentionally unbound):
|
||||
- GET /health -> {status:"healthy"} after options validation binds.
|
||||
- GET /ready -> MongoDB ping; 503 on MongoException/Timeout.
|
||||
- GET /jobs?kind=&limit= -> recent runs.
|
||||
- GET /jobs/{id} -> run detail.
|
||||
- GET /jobs/definitions -> definitions with lastRun.
|
||||
- GET /jobs/definitions/{kind} -> definition + lastRun or 404.
|
||||
- GET /jobs/definitions/{kind}/runs?limit= -> recent runs or 404 if kind unknown.
|
||||
- GET /jobs/active -> currently running.
|
||||
- POST /jobs/{*jobKind} with {trigger?,parameters?} -> 202 Accepted (Location:/jobs/{runId}) | 404 | 409 | 423.
|
||||
# AGENTS
|
||||
## Role
|
||||
Minimal API host wiring configuration, storage, plugin routines, and job endpoints. Operational surface for health, readiness, and job control.
|
||||
## Scope
|
||||
- Configuration: appsettings.json + etc/concelier.yaml (yaml path = ../etc/concelier.yaml); bind into ConcelierOptions with validation (Only Mongo supported).
|
||||
- Mongo: MongoUrl from options.Storage.Dsn; IMongoClient/IMongoDatabase singletons; default database name fallback (options -> URL -> "concelier").
|
||||
- Services: AddMongoStorage(); AddSourceHttpClients(); RegisterPluginRoutines(configuration, PluginHostOptions).
|
||||
- Bootstrap: MongoBootstrapper.InitializeAsync on startup.
|
||||
- Endpoints (configuration & job control only; root path intentionally unbound):
|
||||
- GET /health -> {status:"healthy"} after options validation binds.
|
||||
- GET /ready -> MongoDB ping; 503 on MongoException/Timeout.
|
||||
- GET /jobs?kind=&limit= -> recent runs.
|
||||
- GET /jobs/{id} -> run detail.
|
||||
- GET /jobs/definitions -> definitions with lastRun.
|
||||
- GET /jobs/definitions/{kind} -> definition + lastRun or 404.
|
||||
- GET /jobs/definitions/{kind}/runs?limit= -> recent runs or 404 if kind unknown.
|
||||
- GET /jobs/active -> currently running.
|
||||
- POST /jobs/{*jobKind} with {trigger?,parameters?} -> 202 Accepted (Location:/jobs/{runId}) | 404 | 409 | 423.
|
||||
- PluginHost defaults: BaseDirectory = solution root; PluginsDirectory = "StellaOps.Concelier.PluginBinaries"; SearchPatterns += "StellaOps.Concelier.Plugin.*.dll"; EnsureDirectoryExists = true.
|
||||
## Participants
|
||||
- Core job system; Storage.Mongo; Source.Common HTTP clients; Exporter and Connector plugin routines discover/register jobs.
|
||||
## Interfaces & contracts
|
||||
- Dependency injection boundary for all connectors/exporters; IOptions<ConcelierOptions> validated on start.
|
||||
- Cancellation: pass app.Lifetime.ApplicationStopping to bootstrapper.
|
||||
## In/Out of scope
|
||||
In: hosting, DI composition, REST surface, readiness checks.
|
||||
Out: business logic of jobs, HTML UI, authn/z (future).
|
||||
## Observability & security expectations
|
||||
- Log startup config (redact DSN credentials), plugin scan results (missing ordered plugins if any).
|
||||
- Structured responses with status codes; no stack traces in HTTP bodies; errors mapped cleanly.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.WebService.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
## Participants
|
||||
- Core job system; Storage.Mongo; Source.Common HTTP clients; Exporter and Connector plugin routines discover/register jobs.
|
||||
## Interfaces & contracts
|
||||
- Dependency injection boundary for all connectors/exporters; IOptions<ConcelierOptions> validated on start.
|
||||
- Cancellation: pass app.Lifetime.ApplicationStopping to bootstrapper.
|
||||
## In/Out of scope
|
||||
In: hosting, DI composition, REST surface, readiness checks.
|
||||
Out: business logic of jobs, HTML UI, authn/z (future).
|
||||
## Observability & security expectations
|
||||
- Log startup config (redact DSN credentials), plugin scan results (missing ordered plugins if any).
|
||||
- Structured responses with status codes; no stack traces in HTTP bodies; errors mapped cleanly.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.WebService.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,40 +1,51 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Bootstrap the ACSC (Australian Cyber Security Centre) advisories connector so the Concelier pipeline can ingest, normalise, and enrich ACSC security bulletins.
|
||||
|
||||
## Scope
|
||||
- Research the authoritative ACSC advisory feed (RSS/Atom, JSON API, or HTML).
|
||||
- Implement fetch windowing, cursor persistence, and retry strategy consistent with other external connectors.
|
||||
- Parse advisory content (summary, affected products, mitigation guidance, references).
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected packages, and provenance metadata.
|
||||
- Provide deterministic fixtures and regression tests that cover fetch/parse/map flows.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` for HTTP client creation, fetch service, and DTO persistence helpers.
|
||||
- `Storage.Mongo` for raw/document/DTO/advisory storage plus cursor management.
|
||||
- `Concelier.Models` for canonical advisory structures and provenance utilities.
|
||||
- `Concelier.Testing` for integration harnesses and snapshot helpers.
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds should follow the pattern `acsc:fetch`, `acsc:parse`, `acsc:map`.
|
||||
- Documents persisted to Mongo must include ETag/Last-Modified metadata when the source exposes it.
|
||||
- Canonical advisories must emit aliases (ACSC ID + CVE IDs) and references (official bulletin + vendor notices).
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Initial end-to-end connector implementation with tests, fixtures, and range primitive coverage.
|
||||
- Minimal telemetry (logging + diagnostics counters) consistent with other connectors.
|
||||
|
||||
Out of scope:
|
||||
- Upstream remediation automation or vendor-specific enrichment beyond ACSC data.
|
||||
- Export-related changes (handled by exporter teams).
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log key lifecycle events (fetch/page processed, parse success/error counts, mapping stats).
|
||||
- Sanitise HTML safely and avoid persisting external scripts or embedded media.
|
||||
- Handle transient fetch failures gracefully with exponential backoff and mark failures in source state.
|
||||
|
||||
## Tests
|
||||
- Add integration-style tests under `StellaOps.Concelier.Connector.Acsc.Tests` covering fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; provide UPDATE flag flow for regeneration.
|
||||
- Validate determinism (ordering, casing, timestamps) to satisfy pipeline reproducibility requirements.
|
||||
# AGENTS
|
||||
## Role
|
||||
Bootstrap the ACSC (Australian Cyber Security Centre) advisories connector so the Concelier pipeline can ingest, normalise, and enrich ACSC security bulletins.
|
||||
|
||||
## Scope
|
||||
- Research the authoritative ACSC advisory feed (RSS/Atom, JSON API, or HTML).
|
||||
- Implement fetch windowing, cursor persistence, and retry strategy consistent with other external connectors.
|
||||
- Parse advisory content (summary, affected products, mitigation guidance, references).
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected packages, and provenance metadata.
|
||||
- Provide deterministic fixtures and regression tests that cover fetch/parse/map flows.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` for HTTP client creation, fetch service, and DTO persistence helpers.
|
||||
- `Storage.Mongo` for raw/document/DTO/advisory storage plus cursor management.
|
||||
- `Concelier.Models` for canonical advisory structures and provenance utilities.
|
||||
- `Concelier.Testing` for integration harnesses and snapshot helpers.
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds should follow the pattern `acsc:fetch`, `acsc:parse`, `acsc:map`.
|
||||
- Documents persisted to Mongo must include ETag/Last-Modified metadata when the source exposes it.
|
||||
- Canonical advisories must emit aliases (ACSC ID + CVE IDs) and references (official bulletin + vendor notices).
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Initial end-to-end connector implementation with tests, fixtures, and range primitive coverage.
|
||||
- Minimal telemetry (logging + diagnostics counters) consistent with other connectors.
|
||||
|
||||
Out of scope:
|
||||
- Upstream remediation automation or vendor-specific enrichment beyond ACSC data.
|
||||
- Export-related changes (handled by exporter teams).
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log key lifecycle events (fetch/page processed, parse success/error counts, mapping stats).
|
||||
- Sanitise HTML safely and avoid persisting external scripts or embedded media.
|
||||
- Handle transient fetch failures gracefully with exponential backoff and mark failures in source state.
|
||||
|
||||
## Tests
|
||||
- Add integration-style tests under `StellaOps.Concelier.Connector.Acsc.Tests` covering fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; provide UPDATE flag flow for regeneration.
|
||||
- Validate determinism (ordering, casing, timestamps) to satisfy pipeline reproducibility requirements.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,40 +1,51 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Build the CCCS (Canadian Centre for Cyber Security) advisories connector so Concelier can ingest national cyber bulletins alongside other vendor/regional sources.
|
||||
|
||||
## Scope
|
||||
- Research CCCS advisory feeds (RSS/Atom, JSON API, or HTML listings) and define the canonical fetch workflow.
|
||||
- Implement fetch, parse, and mapping stages with deterministic cursoring and retry/backoff behaviour.
|
||||
- Normalise advisory content (summary, affected vendors/products, mitigation guidance, references, CVE IDs).
|
||||
- Emit canonical `Advisory` records with aliases, references, affected packages, and provenance metadata.
|
||||
- Provide fixtures and regression tests to keep the connector deterministic.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP clients, fetch service, DTO storage helpers).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores + source state).
|
||||
- `Concelier.Models` (canonical advisory data structures).
|
||||
- `Concelier.Testing` (integration fixtures and snapshot utilities).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `cccs:fetch`, `cccs:parse`, `cccs:map`.
|
||||
- Persist ETag/Last-Modified metadata when the upstream supports it.
|
||||
- Include alias entries for CCCS advisory IDs plus referenced CVE IDs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- End-to-end connector implementation with range primitive coverage for affected packages.
|
||||
- Minimal telemetry logging/counters matching other connectors.
|
||||
|
||||
Out of scope:
|
||||
- Automated remediation actions or vendor-specific enrichment beyond CCCS published data.
|
||||
- Export or downstream pipeline changes.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch attempts, success/failure counts, and mapping statistics.
|
||||
- Sanitize HTML safely, dropping scripts/styles before storing DTOs.
|
||||
- Respect upstream rate limits; mark failures in source state with backoff.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Cccs.Tests` covering fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Validate deterministic ordering and timestamps to maintain reproducibility.
|
||||
# AGENTS
|
||||
## Role
|
||||
Build the CCCS (Canadian Centre for Cyber Security) advisories connector so Concelier can ingest national cyber bulletins alongside other vendor/regional sources.
|
||||
|
||||
## Scope
|
||||
- Research CCCS advisory feeds (RSS/Atom, JSON API, or HTML listings) and define the canonical fetch workflow.
|
||||
- Implement fetch, parse, and mapping stages with deterministic cursoring and retry/backoff behaviour.
|
||||
- Normalise advisory content (summary, affected vendors/products, mitigation guidance, references, CVE IDs).
|
||||
- Emit canonical `Advisory` records with aliases, references, affected packages, and provenance metadata.
|
||||
- Provide fixtures and regression tests to keep the connector deterministic.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP clients, fetch service, DTO storage helpers).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores + source state).
|
||||
- `Concelier.Models` (canonical advisory data structures).
|
||||
- `Concelier.Testing` (integration fixtures and snapshot utilities).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `cccs:fetch`, `cccs:parse`, `cccs:map`.
|
||||
- Persist ETag/Last-Modified metadata when the upstream supports it.
|
||||
- Include alias entries for CCCS advisory IDs plus referenced CVE IDs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- End-to-end connector implementation with range primitive coverage for affected packages.
|
||||
- Minimal telemetry logging/counters matching other connectors.
|
||||
|
||||
Out of scope:
|
||||
- Automated remediation actions or vendor-specific enrichment beyond CCCS published data.
|
||||
- Export or downstream pipeline changes.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch attempts, success/failure counts, and mapping statistics.
|
||||
- Sanitize HTML safely, dropping scripts/styles before storing DTOs.
|
||||
- Respect upstream rate limits; mark failures in source state with backoff.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Cccs.Tests` covering fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Validate deterministic ordering and timestamps to maintain reproducibility.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,40 +1,51 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Deliver a connector for Germany’s CERT-Bund advisories so Concelier can ingest, normalise, and enrich BSI alerts alongside other national feeds.
|
||||
|
||||
## Scope
|
||||
- Identify the authoritative CERT-Bund advisory feed(s) (RSS/Atom, JSON, CSV, or HTML).
|
||||
- Implement fetch/cursor logic with proper windowing, dedupe, and failure backoff.
|
||||
- Parse advisory detail pages for summary, affected products/vendors, mitigation, and references.
|
||||
- Map advisories into canonical `Advisory` objects including aliases, references, affected packages, and provenance/range primitives.
|
||||
- Provide deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical data model).
|
||||
- `Concelier.Testing` (integration harness, snapshot utilities).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `certbund:fetch`, `certbund:parse`, `certbund:map`.
|
||||
- Persist upstream metadata (ETag/Last-Modified) if provided.
|
||||
- Alias set should include CERT-Bund ID and referenced CVE entries.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- End-to-end connector implementation with deterministic tests and range primitive coverage.
|
||||
- Baseline logging/metrics for pipeline observability.
|
||||
|
||||
Out of scope:
|
||||
- Non-advisory CERT-Bund digests or newsletters.
|
||||
- Downstream exporter changes.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch attempts, item counts, and mapping metrics.
|
||||
- Sanitize HTML thoroughly before persistence.
|
||||
- Handle transient failures gracefully with exponential backoff and failure records in source state.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.CertBund.Tests` covering fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support regeneration via environment flag.
|
||||
- Ensure deterministic ordering, casing, and timestamps.
|
||||
# AGENTS
|
||||
## Role
|
||||
Deliver a connector for Germany’s CERT-Bund advisories so Concelier can ingest, normalise, and enrich BSI alerts alongside other national feeds.
|
||||
|
||||
## Scope
|
||||
- Identify the authoritative CERT-Bund advisory feed(s) (RSS/Atom, JSON, CSV, or HTML).
|
||||
- Implement fetch/cursor logic with proper windowing, dedupe, and failure backoff.
|
||||
- Parse advisory detail pages for summary, affected products/vendors, mitigation, and references.
|
||||
- Map advisories into canonical `Advisory` objects including aliases, references, affected packages, and provenance/range primitives.
|
||||
- Provide deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical data model).
|
||||
- `Concelier.Testing` (integration harness, snapshot utilities).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `certbund:fetch`, `certbund:parse`, `certbund:map`.
|
||||
- Persist upstream metadata (ETag/Last-Modified) if provided.
|
||||
- Alias set should include CERT-Bund ID and referenced CVE entries.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- End-to-end connector implementation with deterministic tests and range primitive coverage.
|
||||
- Baseline logging/metrics for pipeline observability.
|
||||
|
||||
Out of scope:
|
||||
- Non-advisory CERT-Bund digests or newsletters.
|
||||
- Downstream exporter changes.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch attempts, item counts, and mapping metrics.
|
||||
- Sanitize HTML thoroughly before persistence.
|
||||
- Handle transient failures gracefully with exponential backoff and failure records in source state.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.CertBund.Tests` covering fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support regeneration via environment flag.
|
||||
- Ensure deterministic ordering, casing, and timestamps.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,38 +1,49 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the CERT/CC (Carnegie Mellon CERT Coordination Center) advisory connector so Concelier can ingest US CERT coordination bulletins.
|
||||
|
||||
## Scope
|
||||
- Identify CERT/CC advisory publication format (VU#, blog, RSS, JSON) and define fetch cadence/windowing.
|
||||
- Implement fetch, parse, and mapping jobs with cursor persistence and dedupe.
|
||||
- Normalise advisory content (summary, impacted vendors, products, recommended mitigations, CVEs).
|
||||
- Produce canonical `Advisory` objects including aliases, references, affected packages, and range primitive metadata.
|
||||
- Supply fixtures and deterministic regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores and state).
|
||||
- `Concelier.Models` (canonical structures).
|
||||
- `Concelier.Testing` (integration tests and snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `certcc:fetch`, `certcc:parse`, `certcc:map`.
|
||||
- Persist upstream caching metadata (ETag/Last-Modified) when available.
|
||||
- Aliases should capture CERT/CC VU IDs and referenced CVEs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- End-to-end connector with range primitive instrumentation and telemetry.
|
||||
|
||||
Out of scope:
|
||||
- ICS-CERT alerts (handled by dedicated connector) or blog posts unrelated to advisories.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch and mapping statistics; surface failures with backoff.
|
||||
- Sanitise HTML sources before persistence.
|
||||
- Respect upstream throttling via retry/backoff.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.CertCc.Tests` to cover fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories and support UPDATE flag for regeneration.
|
||||
- Ensure deterministic ordering and timestamp normalisation.
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the CERT/CC (Carnegie Mellon CERT Coordination Center) advisory connector so Concelier can ingest US CERT coordination bulletins.
|
||||
|
||||
## Scope
|
||||
- Identify CERT/CC advisory publication format (VU#, blog, RSS, JSON) and define fetch cadence/windowing.
|
||||
- Implement fetch, parse, and mapping jobs with cursor persistence and dedupe.
|
||||
- Normalise advisory content (summary, impacted vendors, products, recommended mitigations, CVEs).
|
||||
- Produce canonical `Advisory` objects including aliases, references, affected packages, and range primitive metadata.
|
||||
- Supply fixtures and deterministic regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores and state).
|
||||
- `Concelier.Models` (canonical structures).
|
||||
- `Concelier.Testing` (integration tests and snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `certcc:fetch`, `certcc:parse`, `certcc:map`.
|
||||
- Persist upstream caching metadata (ETag/Last-Modified) when available.
|
||||
- Aliases should capture CERT/CC VU IDs and referenced CVEs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- End-to-end connector with range primitive instrumentation and telemetry.
|
||||
|
||||
Out of scope:
|
||||
- ICS-CERT alerts (handled by dedicated connector) or blog posts unrelated to advisories.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch and mapping statistics; surface failures with backoff.
|
||||
- Sanitise HTML sources before persistence.
|
||||
- Respect upstream throttling via retry/backoff.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.CertCc.Tests` to cover fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories and support UPDATE flag for regeneration.
|
||||
- Ensure deterministic ordering and timestamp normalisation.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,27 +1,38 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
ANSSI CERT-FR advisories connector (avis/alertes) providing national enrichment: advisory metadata, CVE links, mitigation notes, and references.
|
||||
## Scope
|
||||
- Harvest CERT-FR items via RSS and/or list pages; follow item pages for detail; window by publish/update date.
|
||||
- Validate HTML or JSON payloads; extract structured fields; map to canonical aliases, references, severity text.
|
||||
- Maintain watermarks and de-duplication by content hash; idempotent processing.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML parsing helpers, validators).
|
||||
- Storage.Mongo (document, dto, advisory, reference, source_state).
|
||||
- Models (canonical).
|
||||
- Core/WebService (jobs: source:certfr:fetch|parse|map).
|
||||
- Merge engine (later) to enrich only.
|
||||
## Interfaces & contracts
|
||||
- Treat CERT-FR as enrichment; never override distro or PSIRT version ranges absent concrete evidence.
|
||||
- References must include primary bulletin URL and vendor links; tag kind=bulletin/vendor/mitigation appropriately.
|
||||
- Provenance records cite "cert-fr" with method=parser and source URL.
|
||||
## In/Out of scope
|
||||
In: advisory metadata extraction, references, severity text, watermarking.
|
||||
Out: OVAL or package-level authority.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics emits shared `concelier.source.http.*` counters/histograms tagged `concelier.source=certfr`, covering fetch counts, parse failures, and map activity.
|
||||
- Logs: feed URL(s), item ids/urls, extraction durations; no PII; allowlist hostnames.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.CertFr.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
ANSSI CERT-FR advisories connector (avis/alertes) providing national enrichment: advisory metadata, CVE links, mitigation notes, and references.
|
||||
## Scope
|
||||
- Harvest CERT-FR items via RSS and/or list pages; follow item pages for detail; window by publish/update date.
|
||||
- Validate HTML or JSON payloads; extract structured fields; map to canonical aliases, references, severity text.
|
||||
- Maintain watermarks and de-duplication by content hash; idempotent processing.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML parsing helpers, validators).
|
||||
- Storage.Mongo (document, dto, advisory, reference, source_state).
|
||||
- Models (canonical).
|
||||
- Core/WebService (jobs: source:certfr:fetch|parse|map).
|
||||
- Merge engine (later) to enrich only.
|
||||
## Interfaces & contracts
|
||||
- Treat CERT-FR as enrichment; never override distro or PSIRT version ranges absent concrete evidence.
|
||||
- References must include primary bulletin URL and vendor links; tag kind=bulletin/vendor/mitigation appropriately.
|
||||
- Provenance records cite "cert-fr" with method=parser and source URL.
|
||||
## In/Out of scope
|
||||
In: advisory metadata extraction, references, severity text, watermarking.
|
||||
Out: OVAL or package-level authority.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics emits shared `concelier.source.http.*` counters/histograms tagged `concelier.source=certfr`, covering fetch counts, parse failures, and map activity.
|
||||
- Logs: feed URL(s), item ids/urls, extraction durations; no PII; allowlist hostnames.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.CertFr.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,28 +1,39 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
CERT-In national CERT connector; enrichment advisories for India; maps CVE lists, advisory text, mitigations, and references; non-authoritative for package ranges unless explicit evidence is present.
|
||||
## Scope
|
||||
- Discover and fetch advisories from the CERT-In portal; window by advisory code/date; follow detail pages.
|
||||
- Validate HTML or JSON; extract title, summary, CVEs, affected vendor names, mitigations; map references; normalize dates and IDs.
|
||||
- Persist raw docs and maintain source_state cursor; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML parsing, normalization, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, reference, source_state).
|
||||
- Models (canonical).
|
||||
- Core/WebService (jobs: source:certin:fetch|parse|map).
|
||||
- Merge engine treats CERT-In as enrichment (no override of PSIRT or OVAL without concrete ranges).
|
||||
## Interfaces & contracts
|
||||
- Aliases: advisory code if stable (scheme "CERT-IN") and CVE ids; if code is not stable, store as reference only.
|
||||
- References typed: bulletin/advisory/vendor/mitigation; deduped.
|
||||
- Affected omitted unless CERT-In publishes explicit version or fix details.
|
||||
- Provenance: method=parser; value=advisory code or URL; recordedAt.
|
||||
## In/Out of scope
|
||||
In: enrichment, aliasing where stable, references, mitigation text.
|
||||
Out: package range authority; scraping behind auth walls.
|
||||
## Observability & security expectations
|
||||
- Metrics: shared `concelier.source.http.*` counters/histograms from SourceDiagnostics tagged `concelier.source=certin` capture fetch volume, parse failures, and map enrich counts.
|
||||
- Logs: advisory codes, CVE counts per advisory, timing; allowlist host; redact personal data if present.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.CertIn.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
CERT-In national CERT connector; enrichment advisories for India; maps CVE lists, advisory text, mitigations, and references; non-authoritative for package ranges unless explicit evidence is present.
|
||||
## Scope
|
||||
- Discover and fetch advisories from the CERT-In portal; window by advisory code/date; follow detail pages.
|
||||
- Validate HTML or JSON; extract title, summary, CVEs, affected vendor names, mitigations; map references; normalize dates and IDs.
|
||||
- Persist raw docs and maintain source_state cursor; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML parsing, normalization, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, reference, source_state).
|
||||
- Models (canonical).
|
||||
- Core/WebService (jobs: source:certin:fetch|parse|map).
|
||||
- Merge engine treats CERT-In as enrichment (no override of PSIRT or OVAL without concrete ranges).
|
||||
## Interfaces & contracts
|
||||
- Aliases: advisory code if stable (scheme "CERT-IN") and CVE ids; if code is not stable, store as reference only.
|
||||
- References typed: bulletin/advisory/vendor/mitigation; deduped.
|
||||
- Affected omitted unless CERT-In publishes explicit version or fix details.
|
||||
- Provenance: method=parser; value=advisory code or URL; recordedAt.
|
||||
## In/Out of scope
|
||||
In: enrichment, aliasing where stable, references, mitigation text.
|
||||
Out: package range authority; scraping behind auth walls.
|
||||
## Observability & security expectations
|
||||
- Metrics: shared `concelier.source.http.*` counters/histograms from SourceDiagnostics tagged `concelier.source=certin` capture fetch volume, parse failures, and map enrich counts.
|
||||
- Logs: advisory codes, CVE counts per advisory, timing; allowlist host; redact personal data if present.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.CertIn.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,31 +1,42 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Shared connector toolkit. Provides HTTP clients, retry/backoff, conditional GET (ETag/Last-Modified), schema validation, pagination helpers, clocks, and common DTO utilities for all connectors.
|
||||
## Scope
|
||||
- Typed HttpClient registrations with allowlisted hosts and timeouts.
|
||||
- Request pipeline: retries with jitter, backoff on 429/5xx, rate-limit tracking per source.
|
||||
- Conditional GET helpers (If-None-Match, If-Modified-Since), window cursors, and pagination iterators.
|
||||
- Validators: JSON Schema, XML Schema (for example XmlSchemaValidator), and sanitizers.
|
||||
- Content hashing and raw document capture helpers; metadata extraction (headers, status).
|
||||
- HTML sanitization, URL normalization, and PDF-to-text extraction utilities for feeds that require cleanup before validation.
|
||||
## Participants
|
||||
- Source.* connectors (NVD, Red Hat, JVN, PSIRTs, CERTs, ICS).
|
||||
- Storage.Mongo (document/dto repositories using shared shapes).
|
||||
- Core (jobs schedule/trigger for connectors).
|
||||
- QA (canned HTTP server harness, schema fixtures).
|
||||
## Interfaces & contracts
|
||||
- All network calls must pass through configured HttpClient with allowlist and sane timeouts; no direct new HttpClient().
|
||||
- Validators return detailed errors; invalid payloads quarantined and not mapped.
|
||||
- Cursor helpers implement sliding windows and ID-based pagination; rely on IClock/TimeProvider for determinism.
|
||||
- Strict provenance tags for extraction method: parser, oval, package.nevra, llm (gated).
|
||||
## In/Out of scope
|
||||
In: HTTP plumbing, validators, cursor/backoff utilities, hashing.
|
||||
Out: connector-specific schemas/mapping rules, merge precedence.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics publishes `concelier.source.http.*` counters/histograms tagged with `concelier.source=<connector>` plus retries/failures; connector dashboards slice on that tag instead of bespoke metric names.
|
||||
- Logs include uri, status, retries, etag; redact tokens and auth headers.
|
||||
- Distributed tracing hooks and per-connector counters should be wired centrally for consistent observability.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Common.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
Shared connector toolkit. Provides HTTP clients, retry/backoff, conditional GET (ETag/Last-Modified), schema validation, pagination helpers, clocks, and common DTO utilities for all connectors.
|
||||
## Scope
|
||||
- Typed HttpClient registrations with allowlisted hosts and timeouts.
|
||||
- Request pipeline: retries with jitter, backoff on 429/5xx, rate-limit tracking per source.
|
||||
- Conditional GET helpers (If-None-Match, If-Modified-Since), window cursors, and pagination iterators.
|
||||
- Validators: JSON Schema, XML Schema (for example XmlSchemaValidator), and sanitizers.
|
||||
- Content hashing and raw document capture helpers; metadata extraction (headers, status).
|
||||
- HTML sanitization, URL normalization, and PDF-to-text extraction utilities for feeds that require cleanup before validation.
|
||||
## Participants
|
||||
- Source.* connectors (NVD, Red Hat, JVN, PSIRTs, CERTs, ICS).
|
||||
- Storage.Mongo (document/dto repositories using shared shapes).
|
||||
- Core (jobs schedule/trigger for connectors).
|
||||
- QA (canned HTTP server harness, schema fixtures).
|
||||
## Interfaces & contracts
|
||||
- All network calls must pass through configured HttpClient with allowlist and sane timeouts; no direct new HttpClient().
|
||||
- Validators return detailed errors; invalid payloads quarantined and not mapped.
|
||||
- Cursor helpers implement sliding windows and ID-based pagination; rely on IClock/TimeProvider for determinism.
|
||||
- Strict provenance tags for extraction method: parser, oval, package.nevra, llm (gated).
|
||||
## In/Out of scope
|
||||
In: HTTP plumbing, validators, cursor/backoff utilities, hashing.
|
||||
Out: connector-specific schemas/mapping rules, merge precedence.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics publishes `concelier.source.http.*` counters/histograms tagged with `concelier.source=<connector>` plus retries/failures; connector dashboards slice on that tag instead of bespoke metric names.
|
||||
- Logs include uri, status, retries, etag; redact tokens and auth headers.
|
||||
- Distributed tracing hooks and per-connector counters should be wired centrally for consistent observability.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Common.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,38 +1,49 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Create a dedicated CVE connector when we need raw CVE stream ingestion outside of NVD/OSV/National feeds (e.g., CVE JSON 5 API or CNA disclosures).
|
||||
|
||||
## Scope
|
||||
- Determine whether this connector should consume the official CVE JSON 5 API, CNA disclosures, or another stream.
|
||||
- Implement fetch/windowing aligned with CVE publication cadence; manage cursors for incremental backfills.
|
||||
- Parse CVE payloads into DTOs capturing descriptions, affected vendors/products, references, and metrics.
|
||||
- Map CVEs into canonical `Advisory` records (aliases, references, affected packages, range primitives).
|
||||
- Deliver deterministic fixtures/tests for fetch/parse/map lifecycle.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores & source state).
|
||||
- `Concelier.Models` (canonical data model).
|
||||
- `Concelier.Testing` (integration fixtures, snapshot helpers).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `cve:fetch`, `cve:parse`, `cve:map`.
|
||||
- Persist upstream metadata (e.g., `If-Modified-Since`, `cveMetadataDate`) for incremental fetching.
|
||||
- Aliases must include primary CVE ID along with CNA-specific identifiers when available.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Core pipeline for CVE ingestion with provenance/range primitives.
|
||||
|
||||
Out of scope:
|
||||
- Downstream impact scoring or enrichment (handled by other teams).
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch batch sizes, update timestamps, and mapping counts.
|
||||
- Handle rate limits politely with exponential backoff.
|
||||
- Sanitize and validate payloads before persistence.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Cve.Tests` with canned CVE JSON fixtures covering fetch/parse/map.
|
||||
- Snapshot canonical advisories; include env flag for fixture regeneration.
|
||||
- Ensure deterministic ordering and timestamp handling.
|
||||
# AGENTS
|
||||
## Role
|
||||
Create a dedicated CVE connector when we need raw CVE stream ingestion outside of NVD/OSV/National feeds (e.g., CVE JSON 5 API or CNA disclosures).
|
||||
|
||||
## Scope
|
||||
- Determine whether this connector should consume the official CVE JSON 5 API, CNA disclosures, or another stream.
|
||||
- Implement fetch/windowing aligned with CVE publication cadence; manage cursors for incremental backfills.
|
||||
- Parse CVE payloads into DTOs capturing descriptions, affected vendors/products, references, and metrics.
|
||||
- Map CVEs into canonical `Advisory` records (aliases, references, affected packages, range primitives).
|
||||
- Deliver deterministic fixtures/tests for fetch/parse/map lifecycle.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores & source state).
|
||||
- `Concelier.Models` (canonical data model).
|
||||
- `Concelier.Testing` (integration fixtures, snapshot helpers).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `cve:fetch`, `cve:parse`, `cve:map`.
|
||||
- Persist upstream metadata (e.g., `If-Modified-Since`, `cveMetadataDate`) for incremental fetching.
|
||||
- Aliases must include primary CVE ID along with CNA-specific identifiers when available.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Core pipeline for CVE ingestion with provenance/range primitives.
|
||||
|
||||
Out of scope:
|
||||
- Downstream impact scoring or enrichment (handled by other teams).
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch batch sizes, update timestamps, and mapping counts.
|
||||
- Handle rate limits politely with exponential backoff.
|
||||
- Sanitize and validate payloads before persistence.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Cve.Tests` with canned CVE JSON fixtures covering fetch/parse/map.
|
||||
- Snapshot canonical advisories; include env flag for fixture regeneration.
|
||||
- Ensure deterministic ordering and timestamp handling.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,27 +1,38 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Red Hat distro connector (Security Data API and OVAL) providing authoritative OS package ranges (RPM NEVRA) and RHSA metadata; overrides generic registry ranges during merge.
|
||||
## Scope
|
||||
- Fetch Security Data JSON (for example CVRF) via Hydra; window by last_modified or after cursor; optionally ingest OVAL definitions.
|
||||
- Validate payloads; parse advisories, CVEs, affected packages; materialize NEVRA and CPE records.
|
||||
- Map to canonical advisories with affected Type=rpm/cpe, fixedBy NEVRA, RHSA aliasing; persist provenance indicating oval/package.nevra.
|
||||
## Participants
|
||||
- Source.Common (HTTP, throttling, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, source_state).
|
||||
- Models (canonical Affected with NEVRA).
|
||||
- Core/WebService (jobs: source:redhat:fetch|parse|map) already registered.
|
||||
- Merge engine to enforce distro precedence (OVAL or PSIRT greater than NVD).
|
||||
## Interfaces & contracts
|
||||
- Aliases: RHSA-YYYY:NNNN, CVE ids; references include RHSA pages, errata, OVAL links.
|
||||
- Affected: rpm (Identifier=NEVRA key) and cpe entries; versions include introduced/fixed/fixedBy; platforms mark RHEL streams.
|
||||
- Provenance: kind="oval" or "package.nevra" as applicable; value=definition id or package.
|
||||
## In/Out of scope
|
||||
In: authoritative rpm ranges, RHSA mapping, OVAL interpretation, watermarking.
|
||||
Out: building RPM artifacts; cross-distro reconciliation beyond Red Hat.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics publishes `concelier.source.http.*` counters/histograms tagged `concelier.source=redhat`, capturing fetch volumes, parse/OVAL failures, and map affected counts without bespoke metric names.
|
||||
- Logs: cursor bounds, advisory ids, NEVRA counts; allowlist Red Hat endpoints.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Distro.RedHat.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
Red Hat distro connector (Security Data API and OVAL) providing authoritative OS package ranges (RPM NEVRA) and RHSA metadata; overrides generic registry ranges during merge.
|
||||
## Scope
|
||||
- Fetch Security Data JSON (for example CVRF) via Hydra; window by last_modified or after cursor; optionally ingest OVAL definitions.
|
||||
- Validate payloads; parse advisories, CVEs, affected packages; materialize NEVRA and CPE records.
|
||||
- Map to canonical advisories with affected Type=rpm/cpe, fixedBy NEVRA, RHSA aliasing; persist provenance indicating oval/package.nevra.
|
||||
## Participants
|
||||
- Source.Common (HTTP, throttling, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, source_state).
|
||||
- Models (canonical Affected with NEVRA).
|
||||
- Core/WebService (jobs: source:redhat:fetch|parse|map) already registered.
|
||||
- Merge engine to enforce distro precedence (OVAL or PSIRT greater than NVD).
|
||||
## Interfaces & contracts
|
||||
- Aliases: RHSA-YYYY:NNNN, CVE ids; references include RHSA pages, errata, OVAL links.
|
||||
- Affected: rpm (Identifier=NEVRA key) and cpe entries; versions include introduced/fixed/fixedBy; platforms mark RHEL streams.
|
||||
- Provenance: kind="oval" or "package.nevra" as applicable; value=definition id or package.
|
||||
## In/Out of scope
|
||||
In: authoritative rpm ranges, RHSA mapping, OVAL interpretation, watermarking.
|
||||
Out: building RPM artifacts; cross-distro reconciliation beyond Red Hat.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics publishes `concelier.source.http.*` counters/histograms tagged `concelier.source=redhat`, capturing fetch volumes, parse/OVAL failures, and map affected counts without bespoke metric names.
|
||||
- Logs: cursor bounds, advisory ids, NEVRA counts; allowlist Red Hat endpoints.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Distro.RedHat.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# Concelier Ubuntu Connector Charter
|
||||
|
||||
## Mission
|
||||
Implement and maintain the Ubuntu security advisory connector that ingests CVE/USN data into Concelier under the Aggregation-Only Contract (AOC). The connector must capture provenance, version semantics (NEVRA/EVR), and metadata required by downstream policy, export, and AI components while remaining deterministic and offline-friendly.
|
||||
|
||||
## Scope
|
||||
- Connector fetchers/parsers within `StellaOps.Concelier.Connector.Distro.Ubuntu`.
|
||||
- Mirroring support for offline kits (bundle import/export).
|
||||
- Schema updates and fixtures ensuring AOC compliance.
|
||||
- Unit/integration tests validating deterministic ingestion.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/ingestion/aggregation-only-contract.md`
|
||||
- `docs/modules/concelier/operations/connectors/osv.md` (reference style & guardrails)
|
||||
- `docs/modules/concelier/operations/mirror.md` (offline mirroring requirements)
|
||||
- Ubuntu advisory format references linked from sprint notes (tasks should include source URLs).
|
||||
|
||||
## Working Agreement
|
||||
1. **Status sync**: switch task state to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and `TASKS.md` before/after work.
|
||||
2. **AOC adherence**: never derive severity or merge fields; store raw documents with provenance (`source`, `upstream`, `content`, `linkset`, `supersedes`).
|
||||
3. **Deterministic parsing**: normalise timestamps to UTC ISO-8601, sort arrays, stabilise JSON output.
|
||||
4. **Offline readiness**: ensure mirroring path works (no live network unless configured), document bundle usage.
|
||||
5. **Testing**: extend fixtures covering typical, superseding, and edge-case advisories; run connector integration suite.
|
||||
6. **Documentation**: update connector operations docs (add Ubuntu section under `docs/modules/concelier/operations/connectors/`) when formats or configuration change.
|
||||
@@ -1,39 +1,50 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement a connector for GitHub Security Advisories (GHSA) when we need to ingest GHSA content directly (instead of crosswalking via OSV/NVD).
|
||||
|
||||
## Scope
|
||||
- Determine the optimal GHSA data source (GraphQL API, REST, or ecosystem export) and required authentication.
|
||||
- Implement fetch logic with pagination, updated-since filtering, and cursor persistence.
|
||||
- Parse GHSA records (identifiers, summaries, affected packages, versions, references, severity).
|
||||
- Map advisories into canonical `Advisory` objects with aliases, references, affected packages, and range primitives.
|
||||
- Provide deterministic fixtures and regression tests for the full pipeline.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP clients, fetch service, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores and source state).
|
||||
- `Concelier.Models` (canonical advisory types).
|
||||
- `Concelier.Testing` (integration harness, snapshot helpers).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `ghsa:fetch`, `ghsa:parse`, `ghsa:map`.
|
||||
- Support GitHub API authentication & rate limiting (token, retry/backoff).
|
||||
- Alias set must include GHSA IDs and linked CVE IDs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Full GHSA connector implementation with range primitives and provenance instrumentation.
|
||||
|
||||
Out of scope:
|
||||
- Repo-specific advisory ingest (handled via GitHub repo exports).
|
||||
- Downstream ecosystem-specific enrichments.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch pagination, throttling, and mapping stats.
|
||||
- Handle GitHub API rate limits with exponential backoff and `Retry-After`.
|
||||
- Sanitize/validate payloads before persistence.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Ghsa.Tests` with canned GraphQL/REST fixtures.
|
||||
- Snapshot canonical advisories; enable fixture regeneration with env flag.
|
||||
- Confirm deterministic ordering/time normalisation.
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement a connector for GitHub Security Advisories (GHSA) when we need to ingest GHSA content directly (instead of crosswalking via OSV/NVD).
|
||||
|
||||
## Scope
|
||||
- Determine the optimal GHSA data source (GraphQL API, REST, or ecosystem export) and required authentication.
|
||||
- Implement fetch logic with pagination, updated-since filtering, and cursor persistence.
|
||||
- Parse GHSA records (identifiers, summaries, affected packages, versions, references, severity).
|
||||
- Map advisories into canonical `Advisory` objects with aliases, references, affected packages, and range primitives.
|
||||
- Provide deterministic fixtures and regression tests for the full pipeline.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP clients, fetch service, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores and source state).
|
||||
- `Concelier.Models` (canonical advisory types).
|
||||
- `Concelier.Testing` (integration harness, snapshot helpers).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `ghsa:fetch`, `ghsa:parse`, `ghsa:map`.
|
||||
- Support GitHub API authentication & rate limiting (token, retry/backoff).
|
||||
- Alias set must include GHSA IDs and linked CVE IDs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Full GHSA connector implementation with range primitives and provenance instrumentation.
|
||||
|
||||
Out of scope:
|
||||
- Repo-specific advisory ingest (handled via GitHub repo exports).
|
||||
- Downstream ecosystem-specific enrichments.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch pagination, throttling, and mapping stats.
|
||||
- Handle GitHub API rate limits with exponential backoff and `Retry-After`.
|
||||
- Sanitize/validate payloads before persistence.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Ghsa.Tests` with canned GraphQL/REST fixtures.
|
||||
- Snapshot canonical advisories; enable fixture regeneration with env flag.
|
||||
- Confirm deterministic ordering/time normalisation.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,39 +1,50 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the CISA ICS advisory connector to ingest US CISA Industrial Control Systems advisories (distinct from the general CERT feed).
|
||||
|
||||
## Scope
|
||||
- Locate the official CISA ICS advisory feed/API (currently HTML/RSS) and define fetch cadence/windowing.
|
||||
- Build fetch/cursor pipeline with retry/backoff and raw document storage.
|
||||
- Parse advisory content for summary, impacted vendors/products, mitigation, CVEs.
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected ICS packages, and range primitives.
|
||||
- Provide deterministic fixtures and automated regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores + source state).
|
||||
- `Concelier.Models` (canonical advisory structures).
|
||||
- `Concelier.Testing` (integration fixtures and snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `ics-cisa:fetch`, `ics-cisa:parse`, `ics-cisa:map`.
|
||||
- Persist upstream caching metadata (ETag/Last-Modified) when available.
|
||||
- Alias set should include CISA ICS advisory IDs and referenced CVE IDs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- ICS-specific advisories from CISA.
|
||||
- Range primitives capturing vendor/equipment metadata.
|
||||
|
||||
Out of scope:
|
||||
- General CISA alerts (covered elsewhere).
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch attempts, advisory counts, and mapping results.
|
||||
- Sanitize HTML, removing scripts/styles before persistence.
|
||||
- Honour upstream rate limits with exponential backoff.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Ics.Cisa.Tests` to cover fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the CISA ICS advisory connector to ingest US CISA Industrial Control Systems advisories (distinct from the general CERT feed).
|
||||
|
||||
## Scope
|
||||
- Locate the official CISA ICS advisory feed/API (currently HTML/RSS) and define fetch cadence/windowing.
|
||||
- Build fetch/cursor pipeline with retry/backoff and raw document storage.
|
||||
- Parse advisory content for summary, impacted vendors/products, mitigation, CVEs.
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected ICS packages, and range primitives.
|
||||
- Provide deterministic fixtures and automated regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores + source state).
|
||||
- `Concelier.Models` (canonical advisory structures).
|
||||
- `Concelier.Testing` (integration fixtures and snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `ics-cisa:fetch`, `ics-cisa:parse`, `ics-cisa:map`.
|
||||
- Persist upstream caching metadata (ETag/Last-Modified) when available.
|
||||
- Alias set should include CISA ICS advisory IDs and referenced CVE IDs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- ICS-specific advisories from CISA.
|
||||
- Range primitives capturing vendor/equipment metadata.
|
||||
|
||||
Out of scope:
|
||||
- General CISA alerts (covered elsewhere).
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch attempts, advisory counts, and mapping results.
|
||||
- Sanitize HTML, removing scripts/styles before persistence.
|
||||
- Honour upstream rate limits with exponential backoff.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Ics.Cisa.Tests` to cover fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,28 +1,39 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Kaspersky ICS-CERT connector; authoritative for OT/ICS vendor advisories covered by Kaspersky ICS-CERT; maps affected products as ICS domain entities with platform tags.
|
||||
## Scope
|
||||
- Discover/fetch advisories list; window by publish date or slug; fetch detail pages; handle pagination.
|
||||
- Validate HTML or JSON; extract CVEs, affected OT vendors/models/families, mitigations; normalize product taxonomy; map fixed versions if present.
|
||||
- Persist raw docs with sha256; maintain source_state; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML helpers, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, source_state).
|
||||
- Models (canonical; affected.platform="ics-vendor", tags for device families).
|
||||
- Core/WebService (jobs: source:ics-kaspersky:fetch|parse|map).
|
||||
- Merge engine respects ICS vendor authority for OT impact.
|
||||
## Interfaces & contracts
|
||||
- Aliases: CVE ids; if stable ICS-CERT advisory id exists, store scheme "ICS-KASP".
|
||||
- Affected: Type=vendor; Vendor/Product populated; platforms/tags for device family or firmware line; versions with fixedBy when explicit.
|
||||
- References: advisory, vendor pages, mitigation guides; typed; deduped.
|
||||
- Provenance: method=parser; value=advisory slug.
|
||||
## In/Out of scope
|
||||
In: ICS advisory mapping, affected vendor products, mitigation references.
|
||||
Out: firmware downloads; reverse-engineering artifacts.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics publishes `concelier.source.http.*` counters/histograms with `concelier.source=ics-kaspersky` to track fetch totals, parse failures, and mapped affected counts.
|
||||
- Logs: slugs, vendor/product counts, timing; allowlist host.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Ics.Kaspersky.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
Kaspersky ICS-CERT connector; authoritative for OT/ICS vendor advisories covered by Kaspersky ICS-CERT; maps affected products as ICS domain entities with platform tags.
|
||||
## Scope
|
||||
- Discover/fetch advisories list; window by publish date or slug; fetch detail pages; handle pagination.
|
||||
- Validate HTML or JSON; extract CVEs, affected OT vendors/models/families, mitigations; normalize product taxonomy; map fixed versions if present.
|
||||
- Persist raw docs with sha256; maintain source_state; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML helpers, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, source_state).
|
||||
- Models (canonical; affected.platform="ics-vendor", tags for device families).
|
||||
- Core/WebService (jobs: source:ics-kaspersky:fetch|parse|map).
|
||||
- Merge engine respects ICS vendor authority for OT impact.
|
||||
## Interfaces & contracts
|
||||
- Aliases: CVE ids; if stable ICS-CERT advisory id exists, store scheme "ICS-KASP".
|
||||
- Affected: Type=vendor; Vendor/Product populated; platforms/tags for device family or firmware line; versions with fixedBy when explicit.
|
||||
- References: advisory, vendor pages, mitigation guides; typed; deduped.
|
||||
- Provenance: method=parser; value=advisory slug.
|
||||
## In/Out of scope
|
||||
In: ICS advisory mapping, affected vendor products, mitigation references.
|
||||
Out: firmware downloads; reverse-engineering artifacts.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics publishes `concelier.source.http.*` counters/histograms with `concelier.source=ics-kaspersky` to track fetch totals, parse failures, and mapped affected counts.
|
||||
- Logs: slugs, vendor/product counts, timing; allowlist host.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Ics.Kaspersky.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,29 +1,40 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Japan JVN/MyJVN connector; national CERT enrichment with strong identifiers (JVNDB) and vendor status; authoritative only where concrete package evidence exists; otherwise enriches text, severity, references, and aliases.
|
||||
## Scope
|
||||
- Fetch JVNRSS (overview) and VULDEF (detail) via MyJVN API; window by dateFirstPublished/dateLastUpdated; paginate; respect rate limits.
|
||||
- Validate XML or JSON payloads; normalize titles, CVEs, JVNDB ids, vendor status, categories; map references and severity text; attach jp_flags.
|
||||
- Persist raw docs with sha256 and headers; manage source_state cursor; idempotent parse/map.
|
||||
## Participants
|
||||
- Source.Common (HTTP, pagination, XML or XSD validators, retries/backoff).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected (when concrete), reference, jp_flags, source_state).
|
||||
- Models (canonical Advisory/Affected/Provenance).
|
||||
- Core/WebService (jobs: source:jvn:fetch|parse|map).
|
||||
- Merge engine applies enrichment precedence (does not override distro or PSIRT ranges unless JVN gives explicit package truth).
|
||||
## Interfaces & contracts
|
||||
- Aliases include JVNDB-YYYY-NNNNN and CVE ids; scheme "JVNDB".
|
||||
- jp_flags: { jvndb_id, jvn_category, vendor_status }.
|
||||
- References typed: advisory/vendor/bulletin; URLs normalized and deduped.
|
||||
- Affected only when VULDEF gives concrete coordinates; otherwise omit.
|
||||
- Provenance: method=parser; kind=api; value=endpoint plus query window; recordedAt=fetched time.
|
||||
## In/Out of scope
|
||||
In: JVN/MyJVN ingestion, aliases, jp_flags, enrichment mapping, watermarking.
|
||||
Out: overriding distro or PSIRT ranges without concrete evidence; scraping unofficial mirrors.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics emits `concelier.source.http.*` counters/histograms tagged `concelier.source=jvn`, enabling dashboards to track fetch requests, item counts, parse failures, and enrichment/map activity (including jp_flags) via tag filters.
|
||||
- Logs: window bounds, jvndb ids processed, vendor_status distribution; redact API keys.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Jvn.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
Japan JVN/MyJVN connector; national CERT enrichment with strong identifiers (JVNDB) and vendor status; authoritative only where concrete package evidence exists; otherwise enriches text, severity, references, and aliases.
|
||||
## Scope
|
||||
- Fetch JVNRSS (overview) and VULDEF (detail) via MyJVN API; window by dateFirstPublished/dateLastUpdated; paginate; respect rate limits.
|
||||
- Validate XML or JSON payloads; normalize titles, CVEs, JVNDB ids, vendor status, categories; map references and severity text; attach jp_flags.
|
||||
- Persist raw docs with sha256 and headers; manage source_state cursor; idempotent parse/map.
|
||||
## Participants
|
||||
- Source.Common (HTTP, pagination, XML or XSD validators, retries/backoff).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected (when concrete), reference, jp_flags, source_state).
|
||||
- Models (canonical Advisory/Affected/Provenance).
|
||||
- Core/WebService (jobs: source:jvn:fetch|parse|map).
|
||||
- Merge engine applies enrichment precedence (does not override distro or PSIRT ranges unless JVN gives explicit package truth).
|
||||
## Interfaces & contracts
|
||||
- Aliases include JVNDB-YYYY-NNNNN and CVE ids; scheme "JVNDB".
|
||||
- jp_flags: { jvndb_id, jvn_category, vendor_status }.
|
||||
- References typed: advisory/vendor/bulletin; URLs normalized and deduped.
|
||||
- Affected only when VULDEF gives concrete coordinates; otherwise omit.
|
||||
- Provenance: method=parser; kind=api; value=endpoint plus query window; recordedAt=fetched time.
|
||||
## In/Out of scope
|
||||
In: JVN/MyJVN ingestion, aliases, jp_flags, enrichment mapping, watermarking.
|
||||
Out: overriding distro or PSIRT ranges without concrete evidence; scraping unofficial mirrors.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics emits `concelier.source.http.*` counters/histograms tagged `concelier.source=jvn`, enabling dashboards to track fetch requests, item counts, parse failures, and enrichment/map activity (including jp_flags) via tag filters.
|
||||
- Logs: window bounds, jvndb ids processed, vendor_status distribution; redact API keys.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Jvn.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,44 +1,55 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the CISA Known Exploited Vulnerabilities (KEV) catalogue connector to ingest KEV entries for enrichment and policy checks.
|
||||
|
||||
## Scope
|
||||
- Integrate with the official KEV JSON feed; understand schema, update cadence, and pagination (if any).
|
||||
- Implement fetch job with incremental updates, checksum validation, and cursor persistence.
|
||||
- Parse KEV entries (CVE ID, vendor/product, required actions, due dates).
|
||||
- Map entries into canonical `Advisory` (or augmentation) records with aliases, references, affected packages, and range primitives capturing enforcement metadata.
|
||||
- Deliver deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP client, fetch service, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (advisory + range primitive types).
|
||||
- `Concelier.Testing` (integration fixtures & snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `kev:fetch`, `kev:parse`, `kev:map`.
|
||||
- Persist upstream `catalogLastUpdated` / ETag to detect changes.
|
||||
- Alias list must include CVE ID; references should point to CISA KEV listing and vendor advisories.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- KEV feed ingestion and canonical mapping.
|
||||
- Range primitives capturing remediation due dates or vendor requirements.
|
||||
|
||||
Out of scope:
|
||||
- Compliance policy enforcement (handled elsewhere).
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch timestamps, updated entry counts, and mapping stats.
|
||||
- Handle data anomalies and record failures with backoff.
|
||||
- Validate JSON payloads before persistence.
|
||||
- Structured informational logs should surface the catalog version, release timestamp, and advisory counts for each successful parse/map cycle.
|
||||
|
||||
## Operational Notes
|
||||
- HTTP allowlist is limited to `www.cisa.gov`; operators should mirror / proxy that hostname for air-gapped deployments.
|
||||
- CISA publishes KEV updates daily (catalogVersion follows `yyyy.MM.dd`). Expect releases near 16:30–17:00 UTC and retain overlap when scheduling fetches.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Kev.Tests` covering fetch/parse/map with KEV JSON fixtures.
|
||||
- Snapshot canonical output; allow fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the CISA Known Exploited Vulnerabilities (KEV) catalogue connector to ingest KEV entries for enrichment and policy checks.
|
||||
|
||||
## Scope
|
||||
- Integrate with the official KEV JSON feed; understand schema, update cadence, and pagination (if any).
|
||||
- Implement fetch job with incremental updates, checksum validation, and cursor persistence.
|
||||
- Parse KEV entries (CVE ID, vendor/product, required actions, due dates).
|
||||
- Map entries into canonical `Advisory` (or augmentation) records with aliases, references, affected packages, and range primitives capturing enforcement metadata.
|
||||
- Deliver deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP client, fetch service, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (advisory + range primitive types).
|
||||
- `Concelier.Testing` (integration fixtures & snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `kev:fetch`, `kev:parse`, `kev:map`.
|
||||
- Persist upstream `catalogLastUpdated` / ETag to detect changes.
|
||||
- Alias list must include CVE ID; references should point to CISA KEV listing and vendor advisories.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- KEV feed ingestion and canonical mapping.
|
||||
- Range primitives capturing remediation due dates or vendor requirements.
|
||||
|
||||
Out of scope:
|
||||
- Compliance policy enforcement (handled elsewhere).
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch timestamps, updated entry counts, and mapping stats.
|
||||
- Handle data anomalies and record failures with backoff.
|
||||
- Validate JSON payloads before persistence.
|
||||
- Structured informational logs should surface the catalog version, release timestamp, and advisory counts for each successful parse/map cycle.
|
||||
|
||||
## Operational Notes
|
||||
- HTTP allowlist is limited to `www.cisa.gov`; operators should mirror / proxy that hostname for air-gapped deployments.
|
||||
- CISA publishes KEV updates daily (catalogVersion follows `yyyy.MM.dd`). Expect releases near 16:30–17:00 UTC and retain overlap when scheduling fetches.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Kev.Tests` covering fetch/parse/map with KEV JSON fixtures.
|
||||
- Snapshot canonical output; allow fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,38 +1,49 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Deliver the KISA (Korea Internet & Security Agency) advisory connector to ingest Korean vulnerability alerts for Concelier’s regional coverage.
|
||||
|
||||
## Scope
|
||||
- Identify KISA’s advisory feeds (RSS/Atom, JSON, HTML) and determine localisation requirements (Korean language parsing).
|
||||
- Implement fetch/cursor logic with retry/backoff, handling authentication if required.
|
||||
- Parse advisory content to extract summary, affected vendors/products, mitigation steps, CVEs, references.
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected packages, and range primitives (including vendor/language metadata).
|
||||
- Provide deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical data structures).
|
||||
- `Concelier.Testing` (integration fixtures and snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `kisa:fetch`, `kisa:parse`, `kisa:map`.
|
||||
- Persist upstream caching metadata (e.g., ETag/Last-Modified) when available.
|
||||
- Alias set should include KISA advisory identifiers and CVE IDs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Advisory ingestion, translation/normalisation, range primitives.
|
||||
|
||||
Out of scope:
|
||||
- Automated Korean↔English translations beyond summary normalization (unless required for canonical fields).
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch and mapping metrics; record failures with backoff.
|
||||
- Sanitise HTML, removing scripts/styles.
|
||||
- Handle character encoding (UTF-8/Korean) correctly.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Kisa.Tests` covering fetch/parse/map with Korean-language fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
# AGENTS
|
||||
## Role
|
||||
Deliver the KISA (Korea Internet & Security Agency) advisory connector to ingest Korean vulnerability alerts for Concelier’s regional coverage.
|
||||
|
||||
## Scope
|
||||
- Identify KISA’s advisory feeds (RSS/Atom, JSON, HTML) and determine localisation requirements (Korean language parsing).
|
||||
- Implement fetch/cursor logic with retry/backoff, handling authentication if required.
|
||||
- Parse advisory content to extract summary, affected vendors/products, mitigation steps, CVEs, references.
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected packages, and range primitives (including vendor/language metadata).
|
||||
- Provide deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical data structures).
|
||||
- `Concelier.Testing` (integration fixtures and snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `kisa:fetch`, `kisa:parse`, `kisa:map`.
|
||||
- Persist upstream caching metadata (e.g., ETag/Last-Modified) when available.
|
||||
- Alias set should include KISA advisory identifiers and CVE IDs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Advisory ingestion, translation/normalisation, range primitives.
|
||||
|
||||
Out of scope:
|
||||
- Automated Korean↔English translations beyond summary normalization (unless required for canonical fields).
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch and mapping metrics; record failures with backoff.
|
||||
- Sanitise HTML, removing scripts/styles.
|
||||
- Handle character encoding (UTF-8/Korean) correctly.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Kisa.Tests` covering fetch/parse/map with Korean-language fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,26 +1,37 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Connector for NVD API v2: fetch, validate, map CVE items to canonical advisories, including CVSS/CWE/CPE as aliases/references.
|
||||
## Scope
|
||||
- Windowed fetch by modified range (6-12h default) with pagination; respect rate limits.
|
||||
- Parse NVD JSON; validate against schema; extract CVSS v3/v4 metrics, CWE IDs, configurations.cpeMatch.
|
||||
- Map to Advisory: primary id='CVE-YYYY-NNNN'; references; AffectedPackage entries for CPE (type=cpe) and optional vendor tags.
|
||||
- Optional change-history capture: store previous payload hashes and diff summaries for auditing modified CVEs.
|
||||
- Watermark: last successful modified_end; handle partial windows with overlap to avoid misses.
|
||||
## Participants
|
||||
- Merge engine reconciles NVD with PSIRT/OVAL (NVD yields to OVAL for OS packages).
|
||||
- KEV connector may flag some CVEs; NVD severity is preserved but not overridden by KEV.
|
||||
- Exporters consume canonical advisories.
|
||||
## Interfaces & contracts
|
||||
- Job kinds: nvd:fetch, nvd:parse, nvd:map.
|
||||
- Input params: windowHours, since, until; safe defaults in ConcelierOptions.
|
||||
- Output: raw documents, sanitized DTOs, mapped advisories + provenance (document, parser).
|
||||
## In/Out of scope
|
||||
In: registry-level data, references, generic CPEs.
|
||||
Out: authoritative distro package ranges; vendor patch states.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics publishes `concelier.source.http.*` counters/histograms tagged `concelier.source=nvd`; dashboards slice on the tag to track page counts, schema failures, map throughput, and window advancement. Structured logs include window bounds and etag hits.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Nvd.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
Connector for NVD API v2: fetch, validate, map CVE items to canonical advisories, including CVSS/CWE/CPE as aliases/references.
|
||||
## Scope
|
||||
- Windowed fetch by modified range (6-12h default) with pagination; respect rate limits.
|
||||
- Parse NVD JSON; validate against schema; extract CVSS v3/v4 metrics, CWE IDs, configurations.cpeMatch.
|
||||
- Map to Advisory: primary id='CVE-YYYY-NNNN'; references; AffectedPackage entries for CPE (type=cpe) and optional vendor tags.
|
||||
- Optional change-history capture: store previous payload hashes and diff summaries for auditing modified CVEs.
|
||||
- Watermark: last successful modified_end; handle partial windows with overlap to avoid misses.
|
||||
## Participants
|
||||
- Merge engine reconciles NVD with PSIRT/OVAL (NVD yields to OVAL for OS packages).
|
||||
- KEV connector may flag some CVEs; NVD severity is preserved but not overridden by KEV.
|
||||
- Exporters consume canonical advisories.
|
||||
## Interfaces & contracts
|
||||
- Job kinds: nvd:fetch, nvd:parse, nvd:map.
|
||||
- Input params: windowHours, since, until; safe defaults in ConcelierOptions.
|
||||
- Output: raw documents, sanitized DTOs, mapped advisories + provenance (document, parser).
|
||||
## In/Out of scope
|
||||
In: registry-level data, references, generic CPEs.
|
||||
Out: authoritative distro package ranges; vendor patch states.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics publishes `concelier.source.http.*` counters/histograms tagged `concelier.source=nvd`; dashboards slice on the tag to track page counts, schema failures, map throughput, and window advancement. Structured logs include window bounds and etag hits.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Nvd.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,26 +1,37 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Connector for OSV.dev across ecosystems; authoritative SemVer/PURL ranges for OSS packages.
|
||||
## Scope
|
||||
- Fetch by ecosystem or time range; handle pagination and changed-since cursors.
|
||||
- Parse OSV JSON; validate schema; capture introduced/fixed events, database_specific where relevant.
|
||||
- Map to Advisory with AffectedPackage(type=semver, Identifier=PURL); preserve SemVer constraints and introduced/fixed chronology.
|
||||
- Maintain per-ecosystem cursors and deduplicate runs via payload hashes to keep reruns idempotent.
|
||||
## Participants
|
||||
- Source.Common supplies HTTP clients, pagination helpers, and validators.
|
||||
- Storage.Mongo persists documents, DTOs, advisories, and source_state cursors.
|
||||
- Merge engine resolves OSV vs GHSA consistency; prefers SemVer data for libraries; distro OVAL still overrides OS packages.
|
||||
- Exporters serialize per-ecosystem ranges untouched.
|
||||
## Interfaces & contracts
|
||||
- Job kinds: osv:fetch, osv:parse, osv:map (naming consistent with other connectors).
|
||||
- Aliases include CVE/GHSA/OSV IDs; references include advisory/patch/release URLs.
|
||||
- Provenance records method=parser and source=osv.
|
||||
## In/Out of scope
|
||||
In: SemVer+PURL accuracy for OSS ecosystems.
|
||||
Out: vendor PSIRT and distro OVAL specifics.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics exposes the shared `concelier.source.http.*` counters/histograms tagged `concelier.source=osv`; observability dashboards slice on the tag to monitor item volume, schema failures, range counts, and ecosystem coverage. Logs include ecosystem and cursor values.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Osv.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
Connector for OSV.dev across ecosystems; authoritative SemVer/PURL ranges for OSS packages.
|
||||
## Scope
|
||||
- Fetch by ecosystem or time range; handle pagination and changed-since cursors.
|
||||
- Parse OSV JSON; validate schema; capture introduced/fixed events, database_specific where relevant.
|
||||
- Map to Advisory with AffectedPackage(type=semver, Identifier=PURL); preserve SemVer constraints and introduced/fixed chronology.
|
||||
- Maintain per-ecosystem cursors and deduplicate runs via payload hashes to keep reruns idempotent.
|
||||
## Participants
|
||||
- Source.Common supplies HTTP clients, pagination helpers, and validators.
|
||||
- Storage.Mongo persists documents, DTOs, advisories, and source_state cursors.
|
||||
- Merge engine resolves OSV vs GHSA consistency; prefers SemVer data for libraries; distro OVAL still overrides OS packages.
|
||||
- Exporters serialize per-ecosystem ranges untouched.
|
||||
## Interfaces & contracts
|
||||
- Job kinds: osv:fetch, osv:parse, osv:map (naming consistent with other connectors).
|
||||
- Aliases include CVE/GHSA/OSV IDs; references include advisory/patch/release URLs.
|
||||
- Provenance records method=parser and source=osv.
|
||||
## In/Out of scope
|
||||
In: SemVer+PURL accuracy for OSS ecosystems.
|
||||
Out: vendor PSIRT and distro OVAL specifics.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics exposes the shared `concelier.source.http.*` counters/histograms tagged `concelier.source=osv`; observability dashboards slice on the tag to monitor item volume, schema failures, range counts, and ecosystem coverage. Logs include ecosystem and cursor values.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Osv.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,38 +1,49 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the Russian BDU (Vulnerability Database) connector to ingest advisories published by FSTEC’s BDU catalogue.
|
||||
|
||||
## Scope
|
||||
- Determine accessible BDU feeds/APIs (HTML listings, downloadable CSV, SOAP/REST) and access constraints.
|
||||
- Build fetch/cursor pipeline with dedupe, retries, and backoff appropriate for the data source.
|
||||
- Parse advisory records to extract summary, affected vendors/products, mitigation recommendations, CVE IDs.
|
||||
- Map advisories into canonical `Advisory` objects including aliases, references, affected packages, and range primitives.
|
||||
- Provide deterministic fixtures and regression tests for the connector lifecycle.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores + source state).
|
||||
- `Concelier.Models` (canonical data structures).
|
||||
- `Concelier.Testing` (integration harness, snapshot utilities).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `bdu:fetch`, `bdu:parse`, `bdu:map`.
|
||||
- Persist upstream metadata (e.g., record modification timestamp) to drive incremental updates.
|
||||
- Alias set should include BDU identifiers and CVE IDs when present.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Core ingestion/mapping of BDU vulnerability records.
|
||||
|
||||
Out of scope:
|
||||
- Translation beyond normalising required canonical fields.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch/mapping statistics and failure details.
|
||||
- Sanitize source payloads, handling Cyrillic text/encodings correctly.
|
||||
- Respect upstream rate limits and mark failures with backoff.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Ru.Bdu.Tests` covering fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the Russian BDU (Vulnerability Database) connector to ingest advisories published by FSTEC’s BDU catalogue.
|
||||
|
||||
## Scope
|
||||
- Determine accessible BDU feeds/APIs (HTML listings, downloadable CSV, SOAP/REST) and access constraints.
|
||||
- Build fetch/cursor pipeline with dedupe, retries, and backoff appropriate for the data source.
|
||||
- Parse advisory records to extract summary, affected vendors/products, mitigation recommendations, CVE IDs.
|
||||
- Map advisories into canonical `Advisory` objects including aliases, references, affected packages, and range primitives.
|
||||
- Provide deterministic fixtures and regression tests for the connector lifecycle.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores + source state).
|
||||
- `Concelier.Models` (canonical data structures).
|
||||
- `Concelier.Testing` (integration harness, snapshot utilities).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `bdu:fetch`, `bdu:parse`, `bdu:map`.
|
||||
- Persist upstream metadata (e.g., record modification timestamp) to drive incremental updates.
|
||||
- Alias set should include BDU identifiers and CVE IDs when present.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Core ingestion/mapping of BDU vulnerability records.
|
||||
|
||||
Out of scope:
|
||||
- Translation beyond normalising required canonical fields.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch/mapping statistics and failure details.
|
||||
- Sanitize source payloads, handling Cyrillic text/encodings correctly.
|
||||
- Respect upstream rate limits and mark failures with backoff.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Ru.Bdu.Tests` covering fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,38 +1,49 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the Russian NKTsKI (formerly NKCKI) advisories connector to ingest NKTsKI vulnerability bulletins for Concelier’s regional coverage.
|
||||
|
||||
## Scope
|
||||
- Identify NKTsKI advisory feeds/APIs (HTML, RSS, CSV) and access/authentication requirements.
|
||||
- Implement fetch/cursor pipeline with dedupe and failure backoff tailored to the source format.
|
||||
- Parse advisories to extract summary, affected vendors/products, recommended mitigation, and CVE identifiers.
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected packages, and range primitives.
|
||||
- Create deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical data structures).
|
||||
- `Concelier.Testing` (integration fixtures, snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `nkcki:fetch`, `nkcki:parse`, `nkcki:map`.
|
||||
- Persist upstream modification metadata to support incremental updates.
|
||||
- Alias set should include NKTsKI advisory IDs and CVEs when present.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Core ingestion/mapping pipeline with range primitives.
|
||||
|
||||
Out of scope:
|
||||
- Translation beyond canonical field normalisation.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch/mapping activity; mark failures with backoff delays.
|
||||
- Handle Cyrillic text encoding and sanitise HTML safely.
|
||||
- Respect upstream rate limiting/politeness.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Ru.Nkcki.Tests` for fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the Russian NKTsKI (formerly NKCKI) advisories connector to ingest NKTsKI vulnerability bulletins for Concelier’s regional coverage.
|
||||
|
||||
## Scope
|
||||
- Identify NKTsKI advisory feeds/APIs (HTML, RSS, CSV) and access/authentication requirements.
|
||||
- Implement fetch/cursor pipeline with dedupe and failure backoff tailored to the source format.
|
||||
- Parse advisories to extract summary, affected vendors/products, recommended mitigation, and CVE identifiers.
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected packages, and range primitives.
|
||||
- Create deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical data structures).
|
||||
- `Concelier.Testing` (integration fixtures, snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `nkcki:fetch`, `nkcki:parse`, `nkcki:map`.
|
||||
- Persist upstream modification metadata to support incremental updates.
|
||||
- Alias set should include NKTsKI advisory IDs and CVEs when present.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Core ingestion/mapping pipeline with range primitives.
|
||||
|
||||
Out of scope:
|
||||
- Translation beyond canonical field normalisation.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch/mapping activity; mark failures with backoff delays.
|
||||
- Handle Cyrillic text encoding and sanitise HTML safely.
|
||||
- Respect upstream rate limiting/politeness.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Ru.Nkcki.Tests` for fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# Concelier Mirror Connector Charter
|
||||
|
||||
## Mission
|
||||
Provide the connector that ingests advisory mirror bundles produced by Export Center / Mirror Creator into Concelier without external network calls. The connector must preserve bundle provenance, operate under the Aggregation-Only Contract, and support incremental replay for air-gapped deployments.
|
||||
|
||||
## Scope
|
||||
- Connector code in `StellaOps.Concelier.Connector.StellaOpsMirror`.
|
||||
- Bundle validation (signatures, manifests, Merkle roots) and resumable cursor handling.
|
||||
- Integration with mirror ingestion pipelines and Offline Kit workflows.
|
||||
- Tests verifying deterministic behaviour across bundle versions.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/ingestion/aggregation-only-contract.md`
|
||||
- `docs/modules/concelier/operations/mirror.md`
|
||||
- `docs/modules/export-center/architecture.md` (mirror profiles)
|
||||
- `docs/modules/airgap/airgap-mode.md`
|
||||
|
||||
## Working Agreement
|
||||
1. **State updates**: mark tasks `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and `TASKS.md` when work begins/ends.
|
||||
2. **Provenance first**: record bundle identifiers (`bundle_id`, digests, time anchors) alongside every observation/linkset; never mutate raw documents.
|
||||
3. **Deterministic replay**: implement cursor storage and re-run safety (same bundle yields identical outputs).
|
||||
4. **Offline integrity**: validate signatures/hashes before ingest; emit actionable errors for stale/invalid bundles.
|
||||
5. **Testing**: maintain fixtures covering full/delta bundles, supersedes, and failure cases; run integration suite offline.
|
||||
6. **Docs**: update mirror connector guidance in `docs/modules/concelier/operations/mirror.md` whenever workflow changes.
|
||||
@@ -1,28 +1,39 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Adobe PSIRT connector ingesting APSB/APA advisories; authoritative for Adobe products; emits psirt_flags and affected ranges; establishes PSIRT precedence over registry or distro data for Adobe software.
|
||||
## Scope
|
||||
- Discover and fetch APSB/APA index and detail pages; follow product links as needed; window by advisory ID/date.
|
||||
- Validate HTML or JSON; normalize titles, CVE lists, product components, fixed versions/builds; capture mitigation notes and KBs.
|
||||
- Persist raw docs with sha256 and headers; maintain source_state cursors; ensure idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML parsing, retries/backoff, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical Advisory/Affected/Provenance).
|
||||
- Core/WebService (jobs: source:adobe:fetch|parse|map).
|
||||
- Merge engine (later) to apply PSIRT override policy for Adobe packages.
|
||||
## Interfaces & contracts
|
||||
- Aliases include APSB-YYYY-XX (and APA-* when present) plus CVE ids.
|
||||
- Affected entries capture Vendor=Adobe, Product/component names, Type=vendor, Identifier stable (for example product slug), Versions with fixed/fixedBy where available.
|
||||
- References typed: advisory, patch, mitigation, release notes; URLs normalized and deduped.
|
||||
- Provenance.method="parser"; value carries advisory id and URL; recordedAt=fetch time.
|
||||
## In/Out of scope
|
||||
In: PSIRT ingestion, aliases, affected plus fixedBy, psirt_flags, watermark/resume.
|
||||
Out: signing, package artifact downloads, non-Adobe product truth.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics produces `concelier.source.http.*` counters/histograms tagged `concelier.source=adobe`; operators filter on that tag to monitor fetch counts, parse failures, map affected counts, and cursor movement without bespoke metric names.
|
||||
- Logs: advisory ids, product counts, extraction timings; hosts allowlisted; no secret logging.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Adobe.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
Adobe PSIRT connector ingesting APSB/APA advisories; authoritative for Adobe products; emits psirt_flags and affected ranges; establishes PSIRT precedence over registry or distro data for Adobe software.
|
||||
## Scope
|
||||
- Discover and fetch APSB/APA index and detail pages; follow product links as needed; window by advisory ID/date.
|
||||
- Validate HTML or JSON; normalize titles, CVE lists, product components, fixed versions/builds; capture mitigation notes and KBs.
|
||||
- Persist raw docs with sha256 and headers; maintain source_state cursors; ensure idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML parsing, retries/backoff, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical Advisory/Affected/Provenance).
|
||||
- Core/WebService (jobs: source:adobe:fetch|parse|map).
|
||||
- Merge engine (later) to apply PSIRT override policy for Adobe packages.
|
||||
## Interfaces & contracts
|
||||
- Aliases include APSB-YYYY-XX (and APA-* when present) plus CVE ids.
|
||||
- Affected entries capture Vendor=Adobe, Product/component names, Type=vendor, Identifier stable (for example product slug), Versions with fixed/fixedBy where available.
|
||||
- References typed: advisory, patch, mitigation, release notes; URLs normalized and deduped.
|
||||
- Provenance.method="parser"; value carries advisory id and URL; recordedAt=fetch time.
|
||||
## In/Out of scope
|
||||
In: PSIRT ingestion, aliases, affected plus fixedBy, psirt_flags, watermark/resume.
|
||||
Out: signing, package artifact downloads, non-Adobe product truth.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics produces `concelier.source.http.*` counters/histograms tagged `concelier.source=adobe`; operators filter on that tag to monitor fetch counts, parse failures, map affected counts, and cursor movement without bespoke metric names.
|
||||
- Logs: advisory ids, product counts, extraction timings; hosts allowlisted; no secret logging.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Adobe.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,39 +1,50 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the Apple security advisories connector to ingest Apple HT/HT2 security bulletins for macOS/iOS/tvOS/visionOS.
|
||||
|
||||
## Scope
|
||||
- Identify canonical Apple security bulletin feeds (HTML, RSS, JSON) and change detection strategy.
|
||||
- Implement fetch/cursor pipeline with retry/backoff, handling localisation/HTML quirks.
|
||||
- Parse advisories to extract summary, affected products/versions, mitigation, CVEs.
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected packages, and range primitives (SemVer + vendor extensions).
|
||||
- Produce deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical structures + range primitives).
|
||||
- `Concelier.Testing` (integration fixtures/snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `apple:fetch`, `apple:parse`, `apple:map`.
|
||||
- Persist upstream metadata (ETag/Last-Modified or revision IDs) for incremental updates.
|
||||
- Alias set should include Apple HT IDs and CVE IDs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Security advisories covering Apple OS/app updates.
|
||||
- Range primitives capturing device/OS version ranges.
|
||||
|
||||
Out of scope:
|
||||
- Release notes unrelated to security.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch/mapping statistics and failure details.
|
||||
- Sanitize HTML while preserving structured data tables.
|
||||
- Respect upstream rate limits; record failures with backoff.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Vndr.Apple.Tests` covering fetch/parse/map with fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the Apple security advisories connector to ingest Apple HT/HT2 security bulletins for macOS/iOS/tvOS/visionOS.
|
||||
|
||||
## Scope
|
||||
- Identify canonical Apple security bulletin feeds (HTML, RSS, JSON) and change detection strategy.
|
||||
- Implement fetch/cursor pipeline with retry/backoff, handling localisation/HTML quirks.
|
||||
- Parse advisories to extract summary, affected products/versions, mitigation, CVEs.
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected packages, and range primitives (SemVer + vendor extensions).
|
||||
- Produce deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical structures + range primitives).
|
||||
- `Concelier.Testing` (integration fixtures/snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `apple:fetch`, `apple:parse`, `apple:map`.
|
||||
- Persist upstream metadata (ETag/Last-Modified or revision IDs) for incremental updates.
|
||||
- Alias set should include Apple HT IDs and CVE IDs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- Security advisories covering Apple OS/app updates.
|
||||
- Range primitives capturing device/OS version ranges.
|
||||
|
||||
Out of scope:
|
||||
- Release notes unrelated to security.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch/mapping statistics and failure details.
|
||||
- Sanitize HTML while preserving structured data tables.
|
||||
- Respect upstream rate limits; record failures with backoff.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Vndr.Apple.Tests` covering fetch/parse/map with fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,28 +1,39 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Chromium/Chrome vendor feed connector parsing Stable Channel Update posts; authoritative vendor context for Chrome/Chromium versions and CVE lists; maps fixed versions as affected ranges.
|
||||
## Scope
|
||||
- Crawl Chrome Releases blog list; window by publish date; fetch detail posts; identify "Stable Channel Update" and security fix sections.
|
||||
- Validate HTML; extract version trains, platform notes (Windows/macOS/Linux/Android), CVEs, acknowledgements; map fixed versions.
|
||||
- Persist raw docs and maintain source_state cursor; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML helpers, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical; affected ranges by product/version).
|
||||
- Core/WebService (jobs: source:chromium:fetch|parse|map).
|
||||
- Merge engine (later) to respect vendor PSIRT precedence for Chrome.
|
||||
## Interfaces & contracts
|
||||
- Aliases: CHROMIUM-POST:<yyyy-mm-dd or slug> plus CVE ids.
|
||||
- Affected: Vendor=Google, Product=Chrome/Chromium (platform tags), Type=vendor; Versions indicate introduced? (often unknown) and fixed (for example 127.0.6533.88); tags mark platforms.
|
||||
- References: advisory (post URL), release notes, bug links; kind set appropriately.
|
||||
- Provenance: method=parser; value=post slug; recordedAt=fetch time.
|
||||
## In/Out of scope
|
||||
In: vendor advisory mapping, fixed version emission per platform, psirt_flags vendor context.
|
||||
Out: OS distro packaging semantics; bug bounty details beyond references.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics exports the shared `concelier.source.http.*` counters/histograms tagged `concelier.source=chromium`, enabling dashboards to observe fetch volumes, parse failures, and map affected counts via tag filters.
|
||||
- Logs: post slugs, version extracted, platform coverage, timing; allowlist blog host.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Chromium.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
Chromium/Chrome vendor feed connector parsing Stable Channel Update posts; authoritative vendor context for Chrome/Chromium versions and CVE lists; maps fixed versions as affected ranges.
|
||||
## Scope
|
||||
- Crawl Chrome Releases blog list; window by publish date; fetch detail posts; identify "Stable Channel Update" and security fix sections.
|
||||
- Validate HTML; extract version trains, platform notes (Windows/macOS/Linux/Android), CVEs, acknowledgements; map fixed versions.
|
||||
- Persist raw docs and maintain source_state cursor; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, HTML helpers, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical; affected ranges by product/version).
|
||||
- Core/WebService (jobs: source:chromium:fetch|parse|map).
|
||||
- Merge engine (later) to respect vendor PSIRT precedence for Chrome.
|
||||
## Interfaces & contracts
|
||||
- Aliases: CHROMIUM-POST:<yyyy-mm-dd or slug> plus CVE ids.
|
||||
- Affected: Vendor=Google, Product=Chrome/Chromium (platform tags), Type=vendor; Versions indicate introduced? (often unknown) and fixed (for example 127.0.6533.88); tags mark platforms.
|
||||
- References: advisory (post URL), release notes, bug links; kind set appropriately.
|
||||
- Provenance: method=parser; value=post slug; recordedAt=fetch time.
|
||||
## In/Out of scope
|
||||
In: vendor advisory mapping, fixed version emission per platform, psirt_flags vendor context.
|
||||
Out: OS distro packaging semantics; bug bounty details beyond references.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics exports the shared `concelier.source.http.*` counters/histograms tagged `concelier.source=chromium`, enabling dashboards to observe fetch volumes, parse failures, and map affected counts via tag filters.
|
||||
- Logs: post slugs, version extracted, platform coverage, timing; allowlist blog host.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Chromium.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,30 +1,41 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the Cisco security advisory connector to ingest Cisco PSIRT bulletins for Concelier.
|
||||
|
||||
## Scope
|
||||
- Identify Cisco advisory feeds/APIs (XML, HTML, JSON) and define incremental fetch strategy.
|
||||
- Implement fetch/cursor pipeline with retry/backoff and document dedupe.
|
||||
- Parse advisories to extract summary, affected products, Cisco bug IDs, CVEs, mitigation guidance.
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected packages, and range primitives (e.g., SemVer/IOS version metadata).
|
||||
- Provide deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common`, `Storage.Mongo`, `Concelier.Models`, `Concelier.Testing`.
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `cisco:fetch`, `cisco:parse`, `cisco:map`.
|
||||
- Persist upstream metadata (e.g., `Last-Modified`, `advisoryId`).
|
||||
- Alias set should include Cisco advisory IDs, bug IDs, and CVEs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope: Cisco PSIRT advisories, range primitive coverage.
|
||||
Out of scope: Non-security Cisco release notes.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch/mapping statistics, respect Cisco API rate limits, sanitise HTML.
|
||||
- Handle authentication tokens if API requires them.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Vndr.Cisco.Tests` with canned fixtures for fetch/parse/map.
|
||||
- Snapshot canonical advisories and support fixture regeneration.
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the Cisco security advisory connector to ingest Cisco PSIRT bulletins for Concelier.
|
||||
|
||||
## Scope
|
||||
- Identify Cisco advisory feeds/APIs (XML, HTML, JSON) and define incremental fetch strategy.
|
||||
- Implement fetch/cursor pipeline with retry/backoff and document dedupe.
|
||||
- Parse advisories to extract summary, affected products, Cisco bug IDs, CVEs, mitigation guidance.
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected packages, and range primitives (e.g., SemVer/IOS version metadata).
|
||||
- Provide deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common`, `Storage.Mongo`, `Concelier.Models`, `Concelier.Testing`.
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `cisco:fetch`, `cisco:parse`, `cisco:map`.
|
||||
- Persist upstream metadata (e.g., `Last-Modified`, `advisoryId`).
|
||||
- Alias set should include Cisco advisory IDs, bug IDs, and CVEs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope: Cisco PSIRT advisories, range primitive coverage.
|
||||
Out of scope: Non-security Cisco release notes.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch/mapping statistics, respect Cisco API rate limits, sanitise HTML.
|
||||
- Handle authentication tokens if API requires them.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Vndr.Cisco.Tests` with canned fixtures for fetch/parse/map.
|
||||
- Snapshot canonical advisories and support fixture regeneration.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,30 +1,41 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the Microsoft Security Response Center (MSRC) connector to ingest Microsoft security updates (Security Updates API / CVRF).
|
||||
|
||||
## Scope
|
||||
- Identify MSRC data sources (Security Update Guide API, CVRF downloads) and incremental update strategy.
|
||||
- Implement fetch/cursor pipeline with retry/backoff, handling API keys if required.
|
||||
- Parse advisories to extract summary, affected products, KBs, CVEs, severities, mitigations.
|
||||
- Map entries into canonical `Advisory` objects with aliases, references, affected packages, and range primitives (e.g., Windows build numbers, SemVer).
|
||||
- Provide deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common`, `Storage.Mongo`, `Concelier.Models`, `Concelier.Testing`.
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `msrc:fetch`, `msrc:parse`, `msrc:map`.
|
||||
- Persist upstream metadata (e.g., `lastModified`, `releaseDate`).
|
||||
- Alias set should include MSRC ID, CVEs, and KB identifiers.
|
||||
|
||||
## In/Out of scope
|
||||
In scope: Microsoft Security Update Guide advisories.
|
||||
Out of scope: Non-security Microsoft release notes.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch/mapping stats, respect API rate limits, handle authentication securely.
|
||||
- Sanitize payloads; validate JSON/CVRF before persistence.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Vndr.Msrc.Tests` with fixtures covering fetch/parse/map.
|
||||
- Snapshot canonical advisories; support fixture regeneration.
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the Microsoft Security Response Center (MSRC) connector to ingest Microsoft security updates (Security Updates API / CVRF).
|
||||
|
||||
## Scope
|
||||
- Identify MSRC data sources (Security Update Guide API, CVRF downloads) and incremental update strategy.
|
||||
- Implement fetch/cursor pipeline with retry/backoff, handling API keys if required.
|
||||
- Parse advisories to extract summary, affected products, KBs, CVEs, severities, mitigations.
|
||||
- Map entries into canonical `Advisory` objects with aliases, references, affected packages, and range primitives (e.g., Windows build numbers, SemVer).
|
||||
- Provide deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common`, `Storage.Mongo`, `Concelier.Models`, `Concelier.Testing`.
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `msrc:fetch`, `msrc:parse`, `msrc:map`.
|
||||
- Persist upstream metadata (e.g., `lastModified`, `releaseDate`).
|
||||
- Alias set should include MSRC ID, CVEs, and KB identifiers.
|
||||
|
||||
## In/Out of scope
|
||||
In scope: Microsoft Security Update Guide advisories.
|
||||
Out of scope: Non-security Microsoft release notes.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch/mapping stats, respect API rate limits, handle authentication securely.
|
||||
- Sanitize payloads; validate JSON/CVRF before persistence.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Vndr.Msrc.Tests` with fixtures covering fetch/parse/map.
|
||||
- Snapshot canonical advisories; support fixture regeneration.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,27 +1,38 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Oracle PSIRT connector for Critical Patch Updates (CPU) and Security Alerts; authoritative vendor ranges and severities for Oracle products; establishes PSIRT precedence over registry or distro where applicable.
|
||||
## Scope
|
||||
- Harvest CPU calendar pages and per-advisory content; window by CPU cycle (Jan/Apr/Jul/Oct) and last modified timestamps.
|
||||
- Validate HTML or JSON; extract CVE lists, affected products, components, versions, fixed patch levels; map to canonical with aliases and psirt_flags.
|
||||
- Persist raw documents; maintain source_state across cycles; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical; affected ranges for vendor products).
|
||||
- Core/WebService (jobs: source:oracle:fetch|parse|map).
|
||||
- Merge engine (later) to prefer PSIRT ranges over NVD for Oracle products.
|
||||
## Interfaces & contracts
|
||||
- Alias scheme includes CPU:YYYY-QQ plus individual advisory ids when present; include CVE mappings.
|
||||
- Affected entries capture product/component and fixedBy patch version; references include product notes and patch docs; kind=advisory or patch.
|
||||
- Provenance.method=parser; value includes CPU cycle and advisory slug.
|
||||
## In/Out of scope
|
||||
In: PSIRT authoritative mapping, cycles handling, precedence signaling.
|
||||
Out: signing or patch artifact downloads.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics emits `concelier.source.http.*` counters/histograms tagged `concelier.source=oracle`, so observability dashboards slice on that tag to monitor fetch pages, CPU cycle coverage, parse failures, and map affected counts.
|
||||
- Logs: cycle tags, advisory ids, extraction timings; redact nothing sensitive.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Oracle.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
Oracle PSIRT connector for Critical Patch Updates (CPU) and Security Alerts; authoritative vendor ranges and severities for Oracle products; establishes PSIRT precedence over registry or distro where applicable.
|
||||
## Scope
|
||||
- Harvest CPU calendar pages and per-advisory content; window by CPU cycle (Jan/Apr/Jul/Oct) and last modified timestamps.
|
||||
- Validate HTML or JSON; extract CVE lists, affected products, components, versions, fixed patch levels; map to canonical with aliases and psirt_flags.
|
||||
- Persist raw documents; maintain source_state across cycles; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical; affected ranges for vendor products).
|
||||
- Core/WebService (jobs: source:oracle:fetch|parse|map).
|
||||
- Merge engine (later) to prefer PSIRT ranges over NVD for Oracle products.
|
||||
## Interfaces & contracts
|
||||
- Alias scheme includes CPU:YYYY-QQ plus individual advisory ids when present; include CVE mappings.
|
||||
- Affected entries capture product/component and fixedBy patch version; references include product notes and patch docs; kind=advisory or patch.
|
||||
- Provenance.method=parser; value includes CPU cycle and advisory slug.
|
||||
## In/Out of scope
|
||||
In: PSIRT authoritative mapping, cycles handling, precedence signaling.
|
||||
Out: signing or patch artifact downloads.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics emits `concelier.source.http.*` counters/histograms tagged `concelier.source=oracle`, so observability dashboards slice on that tag to monitor fetch pages, CPU cycle coverage, parse failures, and map affected counts.
|
||||
- Logs: cycle tags, advisory ids, extraction timings; redact nothing sensitive.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Oracle.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,28 +1,39 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
VMware/Broadcom PSIRT connector ingesting VMSA advisories; authoritative for VMware products; maps affected versions/builds and emits psirt_flags.
|
||||
## Scope
|
||||
- Discover/fetch VMSA index and detail pages via Broadcom portal; window by advisory ID/date; follow updates/revisions.
|
||||
- Validate HTML or JSON; extract CVEs, affected product versions/builds, workarounds, fixed versions; normalize product naming.
|
||||
- Persist raw docs with sha256; manage source_state; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, cookies/session handling if needed, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical).
|
||||
- Core/WebService (jobs: source:vmware:fetch|parse|map).
|
||||
- Merge engine (later) to prefer PSIRT ranges for VMware products.
|
||||
## Interfaces & contracts
|
||||
- Aliases: VMSA-YYYY-NNNN plus CVEs.
|
||||
- Affected entries include Vendor=VMware, Product plus component; Versions carry fixed/fixedBy; tags may include build numbers or ESXi/VC levels.
|
||||
- References: advisory URL, KBs, workaround pages; typed; deduped.
|
||||
- Provenance: method=parser; value=VMSA id.
|
||||
## In/Out of scope
|
||||
In: PSIRT precedence mapping, affected/fixedBy extraction, advisory references.
|
||||
Out: customer portal authentication flows beyond public advisories; downloading patches.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics emits shared `concelier.source.http.*` counters/histograms tagged `concelier.source=vmware`, allowing dashboards to measure fetch volume, parse failures, and map affected counts without bespoke metric names.
|
||||
- Logs: vmsa ids, product counts, extraction timings; handle portal rate limits politely.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Vmware.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
VMware/Broadcom PSIRT connector ingesting VMSA advisories; authoritative for VMware products; maps affected versions/builds and emits psirt_flags.
|
||||
## Scope
|
||||
- Discover/fetch VMSA index and detail pages via Broadcom portal; window by advisory ID/date; follow updates/revisions.
|
||||
- Validate HTML or JSON; extract CVEs, affected product versions/builds, workarounds, fixed versions; normalize product naming.
|
||||
- Persist raw docs with sha256; manage source_state; idempotent mapping.
|
||||
## Participants
|
||||
- Source.Common (HTTP, cookies/session handling if needed, validators).
|
||||
- Storage.Mongo (document, dto, advisory, alias, affected, reference, psirt_flags, source_state).
|
||||
- Models (canonical).
|
||||
- Core/WebService (jobs: source:vmware:fetch|parse|map).
|
||||
- Merge engine (later) to prefer PSIRT ranges for VMware products.
|
||||
## Interfaces & contracts
|
||||
- Aliases: VMSA-YYYY-NNNN plus CVEs.
|
||||
- Affected entries include Vendor=VMware, Product plus component; Versions carry fixed/fixedBy; tags may include build numbers or ESXi/VC levels.
|
||||
- References: advisory URL, KBs, workaround pages; typed; deduped.
|
||||
- Provenance: method=parser; value=VMSA id.
|
||||
## In/Out of scope
|
||||
In: PSIRT precedence mapping, affected/fixedBy extraction, advisory references.
|
||||
Out: customer portal authentication flows beyond public advisories; downloading patches.
|
||||
## Observability & security expectations
|
||||
- Metrics: SourceDiagnostics emits shared `concelier.source.http.*` counters/histograms tagged `concelier.source=vmware`, allowing dashboards to measure fetch volume, parse failures, and map affected counts without bespoke metric names.
|
||||
- Logs: vmsa ids, product counts, extraction timings; handle portal rate limits politely.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Connector.Vndr.Vmware.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,32 +1,43 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Job orchestration and lifecycle. Registers job definitions, schedules execution, triggers runs, reports status for connectors and exporters.
|
||||
## Scope
|
||||
- Contracts: IJob (execute with CancellationToken), JobRunStatus, JobTriggerOutcome/Result.
|
||||
- Registration: JobSchedulerBuilder.AddJob<T>(kind, cronExpression?, timeout?, leaseDuration?); options recorded in JobSchedulerOptions.
|
||||
- Plugin host integration discovers IJob providers via registered IDependencyInjectionRoutine implementations.
|
||||
- Coordination: start/stop, single-flight via storage locks/leases, run bookkeeping (status, timings, errors).
|
||||
- Triggering: manual/cron/API; parameterized runs; idempotent rejection if already running.
|
||||
- Surfacing: enumerate definitions, last run, recent runs, active runs to WebService endpoints.
|
||||
## Participants
|
||||
- WebService exposes REST endpoints for definitions, runs, active, and trigger.
|
||||
- Storage.Mongo persists job definitions metadata, run documents, and leases (locks collection).
|
||||
- Source connectors and Exporters implement IJob and are registered into the scheduler via DI and Plugin routines.
|
||||
- Models/Merge/Export are invoked indirectly through jobs.
|
||||
- Plugin host runtime loads dependency injection routines that register job definitions.
|
||||
## Interfaces & contracts
|
||||
- Kind naming: family:source:verb (e.g., nvd:fetch, redhat:map, export:trivy-db).
|
||||
- Timeout and lease duration enforce cancellation and duplicate-prevention.
|
||||
- TimeProvider used for deterministic timing in tests.
|
||||
## In/Out of scope
|
||||
In: job lifecycle, registration, trigger semantics, run metadata.
|
||||
Out: business logic of connectors/exporters, HTTP handlers (owned by WebService).
|
||||
## Observability & security expectations
|
||||
- Metrics: job.run.started/succeeded/failed, job.durationMs, job.concurrent.rejected, job.alreadyRunning.
|
||||
- Logs: kind, trigger, params hash, lease holder, outcome; redact params containing secrets.
|
||||
- Honor CancellationToken early and often.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Core.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
# AGENTS
|
||||
## Role
|
||||
Job orchestration and lifecycle. Registers job definitions, schedules execution, triggers runs, reports status for connectors and exporters.
|
||||
## Scope
|
||||
- Contracts: IJob (execute with CancellationToken), JobRunStatus, JobTriggerOutcome/Result.
|
||||
- Registration: JobSchedulerBuilder.AddJob<T>(kind, cronExpression?, timeout?, leaseDuration?); options recorded in JobSchedulerOptions.
|
||||
- Plugin host integration discovers IJob providers via registered IDependencyInjectionRoutine implementations.
|
||||
- Coordination: start/stop, single-flight via storage locks/leases, run bookkeeping (status, timings, errors).
|
||||
- Triggering: manual/cron/API; parameterized runs; idempotent rejection if already running.
|
||||
- Surfacing: enumerate definitions, last run, recent runs, active runs to WebService endpoints.
|
||||
## Participants
|
||||
- WebService exposes REST endpoints for definitions, runs, active, and trigger.
|
||||
- Storage.Mongo persists job definitions metadata, run documents, and leases (locks collection).
|
||||
- Source connectors and Exporters implement IJob and are registered into the scheduler via DI and Plugin routines.
|
||||
- Models/Merge/Export are invoked indirectly through jobs.
|
||||
- Plugin host runtime loads dependency injection routines that register job definitions.
|
||||
## Interfaces & contracts
|
||||
- Kind naming: family:source:verb (e.g., nvd:fetch, redhat:map, export:trivy-db).
|
||||
- Timeout and lease duration enforce cancellation and duplicate-prevention.
|
||||
- TimeProvider used for deterministic timing in tests.
|
||||
## In/Out of scope
|
||||
In: job lifecycle, registration, trigger semantics, run metadata.
|
||||
Out: business logic of connectors/exporters, HTTP handlers (owned by WebService).
|
||||
## Observability & security expectations
|
||||
- Metrics: job.run.started/succeeded/failed, job.durationMs, job.concurrent.rejected, job.alreadyRunning.
|
||||
- Logs: kind, trigger, params hash, lease holder, outcome; redact params containing secrets.
|
||||
- Honor CancellationToken early and often.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Core.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,28 +1,39 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Optional exporter producing vuln-list-shaped JSON tree for downstream trivy-db builder or interoperability. Deterministic, provenance-preserving.
|
||||
## Scope
|
||||
- Transform canonical advisories into directory tree structure mirroring aquasecurity/vuln-list (by ecosystem/vendor/distro as applicable).
|
||||
- Sorting and serialization invariants: stable key order, newline policy, UTC ISO-8601.
|
||||
- Cursoring/incremental export: export_state tracks last advisory hash/time to avoid full rewrites.
|
||||
- Packaging: output directory under exports/json/<timestamp> with reproducible naming; optionally symlink latest.
|
||||
- Optional auxiliary index files (for example severity summaries) may be generated when explicitly requested, but must remain deterministic and avoid altering canonical payloads.
|
||||
## Participants
|
||||
- Storage.Mongo.AdvisoryStore as input; ExportState repository for cursors/digests.
|
||||
- Core scheduler runs JsonExportJob; Plugin DI wires JsonExporter + job.
|
||||
- TrivyDb exporter may consume the rendered tree in v0 (builder path) if configured.
|
||||
## Interfaces & contracts
|
||||
- Job kind: export:json (JsonExportJob).
|
||||
- Determinism: same inputs -> identical file bytes; hash snapshot persisted.
|
||||
- Provenance: include minimal provenance fields when helpful; keep identity stable.
|
||||
## In/Out of scope
|
||||
In: JSON rendering and layout; incremental/deterministic writes.
|
||||
Out: ORAS push and Trivy DB BoltDB writing (owned by Trivy exporter).
|
||||
## Observability & security expectations
|
||||
- Metrics: export.json.records, bytes, duration, delta.changed.
|
||||
- Logs: target path, record counts, digest; no sensitive data.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Exporter.Json.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
# AGENTS
|
||||
## Role
|
||||
Optional exporter producing vuln-list-shaped JSON tree for downstream trivy-db builder or interoperability. Deterministic, provenance-preserving.
|
||||
## Scope
|
||||
- Transform canonical advisories into directory tree structure mirroring aquasecurity/vuln-list (by ecosystem/vendor/distro as applicable).
|
||||
- Sorting and serialization invariants: stable key order, newline policy, UTC ISO-8601.
|
||||
- Cursoring/incremental export: export_state tracks last advisory hash/time to avoid full rewrites.
|
||||
- Packaging: output directory under exports/json/<timestamp> with reproducible naming; optionally symlink latest.
|
||||
- Optional auxiliary index files (for example severity summaries) may be generated when explicitly requested, but must remain deterministic and avoid altering canonical payloads.
|
||||
## Participants
|
||||
- Storage.Mongo.AdvisoryStore as input; ExportState repository for cursors/digests.
|
||||
- Core scheduler runs JsonExportJob; Plugin DI wires JsonExporter + job.
|
||||
- TrivyDb exporter may consume the rendered tree in v0 (builder path) if configured.
|
||||
## Interfaces & contracts
|
||||
- Job kind: export:json (JsonExportJob).
|
||||
- Determinism: same inputs -> identical file bytes; hash snapshot persisted.
|
||||
- Provenance: include minimal provenance fields when helpful; keep identity stable.
|
||||
## In/Out of scope
|
||||
In: JSON rendering and layout; incremental/deterministic writes.
|
||||
Out: ORAS push and Trivy DB BoltDB writing (owned by Trivy exporter).
|
||||
## Observability & security expectations
|
||||
- Metrics: export.json.records, bytes, duration, delta.changed.
|
||||
- Logs: target path, record counts, digest; no sensitive data.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Exporter.Json.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,29 +1,40 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Exporter producing a Trivy-compatible database artifact for self-hosting or offline use. v0: JSON list + metadata; v1: integrate official trivy-db builder or write BoltDB directly; pack and optionally push via ORAS.
|
||||
## Scope
|
||||
- Read canonical advisories; serialize payload for builder or intermediate; write metadata.json (generatedAt, counts).
|
||||
- Output root: exports/trivy/<yyyyMMddHHmmss>; deterministic path components.
|
||||
- OCI/Trivy expectations: layer media type application/vnd.aquasec.trivy.db.layer.v1.tar+gzip; config media type application/vnd.aquasec.trivy.config.v1+json; tag (e.g., 2).
|
||||
- Optional ORAS push; optional offline bundle (db.tar.gz + metadata.json).
|
||||
- DI: TrivyExporter + Jobs.TrivyExportJob registered by TrivyExporterDependencyInjectionRoutine.
|
||||
- Export_state recording: capture digests, counts, start/end timestamps for idempotent reruns and incremental packaging.
|
||||
## Participants
|
||||
- Storage.Mongo.AdvisoryStore as input.
|
||||
- Core scheduler runs export job; WebService/Plugins trigger it.
|
||||
- JSON exporter (optional precursor) if choosing the builder path.
|
||||
## Interfaces & contracts
|
||||
- IFeedExporter.Name = "trivy-db"; ExportAsync(IServiceProvider, CancellationToken).
|
||||
- ConcelierOptions.packaging.trivy governs repo/tag/publish/offline_bundle.
|
||||
- Deterministic sorting and timestamp discipline (UTC; consider build reproducibility knobs).
|
||||
## In/Out of scope
|
||||
In: assembling builder inputs, packing tar.gz, pushing to registry when configured.
|
||||
Out: signing (external pipeline), scanner behavior.
|
||||
## Observability & security expectations
|
||||
- Metrics: export.trivy.records, size_bytes, duration, oras.push.success/fail.
|
||||
- Logs: export path, repo/tag, digest; redact credentials; backoff on push errors.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Exporter.TrivyDb.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
# AGENTS
|
||||
## Role
|
||||
Exporter producing a Trivy-compatible database artifact for self-hosting or offline use. v0: JSON list + metadata; v1: integrate official trivy-db builder or write BoltDB directly; pack and optionally push via ORAS.
|
||||
## Scope
|
||||
- Read canonical advisories; serialize payload for builder or intermediate; write metadata.json (generatedAt, counts).
|
||||
- Output root: exports/trivy/<yyyyMMddHHmmss>; deterministic path components.
|
||||
- OCI/Trivy expectations: layer media type application/vnd.aquasec.trivy.db.layer.v1.tar+gzip; config media type application/vnd.aquasec.trivy.config.v1+json; tag (e.g., 2).
|
||||
- Optional ORAS push; optional offline bundle (db.tar.gz + metadata.json).
|
||||
- DI: TrivyExporter + Jobs.TrivyExportJob registered by TrivyExporterDependencyInjectionRoutine.
|
||||
- Export_state recording: capture digests, counts, start/end timestamps for idempotent reruns and incremental packaging.
|
||||
## Participants
|
||||
- Storage.Mongo.AdvisoryStore as input.
|
||||
- Core scheduler runs export job; WebService/Plugins trigger it.
|
||||
- JSON exporter (optional precursor) if choosing the builder path.
|
||||
## Interfaces & contracts
|
||||
- IFeedExporter.Name = "trivy-db"; ExportAsync(IServiceProvider, CancellationToken).
|
||||
- ConcelierOptions.packaging.trivy governs repo/tag/publish/offline_bundle.
|
||||
- Deterministic sorting and timestamp discipline (UTC; consider build reproducibility knobs).
|
||||
## In/Out of scope
|
||||
In: assembling builder inputs, packing tar.gz, pushing to registry when configured.
|
||||
Out: signing (external pipeline), scanner behavior.
|
||||
## Observability & security expectations
|
||||
- Metrics: export.trivy.records, size_bytes, duration, oras.push.success/fail.
|
||||
- Logs: export path, repo/tag, digest; redact credentials; backoff on push errors.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Exporter.TrivyDb.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,33 +1,44 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Deterministic merge and reconciliation engine; builds identity graph via aliases; applies precedence (PSIRT/OVAL > NVD; KEV flag only; regional feeds enrich); produces canonical advisory JSON and merge_event audit trail.
|
||||
## Scope
|
||||
- Identity: resolve advisory_key (prefer CVE, else PSIRT/Distro/JVN/BDU/GHSA/ICSA); unify aliases; detect collisions.
|
||||
- Precedence: override rules for affected ranges (vendor PSIRT/OVAL over registry), enrichment-only feeds (CERTs/JVN/RU-CERT), KEV toggles exploitKnown only.
|
||||
- Range comparers: RPM NEVRA comparer (epoch:version-release), Debian EVR comparer, SemVer range resolver; platform-aware selection.
|
||||
- Merge algorithm: stable ordering, pure functions, idempotence; compute beforeHash/afterHash over canonical form; write merge_event.
|
||||
- Conflict reporting: counters and logs for identity conflicts, reference merges, range overrides.
|
||||
## Participants
|
||||
- Storage.Mongo (reads raw mapped advisories, writes merged docs plus merge_event).
|
||||
- Models (canonical types).
|
||||
- Exporters (consume merged canonical).
|
||||
- Core/WebService (jobs: merge:run, maybe per-kind).
|
||||
## Interfaces & contracts
|
||||
- AdvisoryMergeService.MergeAsync(ids or byKind): returns summary {processed, merged, overrides, conflicts}.
|
||||
- Precedence table configurable but with sane defaults: RedHat/Ubuntu/Debian/SUSE > Vendor PSIRT > GHSA/OSV > NVD; CERTs enrich; KEV sets flags.
|
||||
- Range selection uses comparers: NevraComparer, DebEvrComparer, SemVerRange; deterministic tie-breakers.
|
||||
- Provenance propagation merges unique entries; references deduped by (url, type).
|
||||
|
||||
## Configuration
|
||||
- Precedence overrides bind via `concelier:merge:precedence:ranks` (dictionary of `source` → `rank`, lower wins). Absent entries fall back to defaults.
|
||||
- Operator workflow: update `etc/concelier.yaml` or environment variables, restart merge job; overrides surface in metrics/logs as `AdvisoryOverride` entries.
|
||||
## In/Out of scope
|
||||
In: merge logic, precedence policy, hashing, event records, comparers.
|
||||
Out: fetching/parsing, exporter packaging, signing.
|
||||
## Observability & security expectations
|
||||
- Metrics: merge.delta.count, merge.identity.conflicts, merge.range.overrides, merge.duration_ms.
|
||||
- Logs: decisions (why replaced), keys involved, hashes; avoid dumping large blobs; redact secrets (none expected).
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Merge.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
# AGENTS
|
||||
## Role
|
||||
Deterministic merge and reconciliation engine; builds identity graph via aliases; applies precedence (PSIRT/OVAL > NVD; KEV flag only; regional feeds enrich); produces canonical advisory JSON and merge_event audit trail.
|
||||
## Scope
|
||||
- Identity: resolve advisory_key (prefer CVE, else PSIRT/Distro/JVN/BDU/GHSA/ICSA); unify aliases; detect collisions.
|
||||
- Precedence: override rules for affected ranges (vendor PSIRT/OVAL over registry), enrichment-only feeds (CERTs/JVN/RU-CERT), KEV toggles exploitKnown only.
|
||||
- Range comparers: RPM NEVRA comparer (epoch:version-release), Debian EVR comparer, SemVer range resolver; platform-aware selection.
|
||||
- Merge algorithm: stable ordering, pure functions, idempotence; compute beforeHash/afterHash over canonical form; write merge_event.
|
||||
- Conflict reporting: counters and logs for identity conflicts, reference merges, range overrides.
|
||||
## Participants
|
||||
- Storage.Mongo (reads raw mapped advisories, writes merged docs plus merge_event).
|
||||
- Models (canonical types).
|
||||
- Exporters (consume merged canonical).
|
||||
- Core/WebService (jobs: merge:run, maybe per-kind).
|
||||
## Interfaces & contracts
|
||||
- AdvisoryMergeService.MergeAsync(ids or byKind): returns summary {processed, merged, overrides, conflicts}.
|
||||
- Precedence table configurable but with sane defaults: RedHat/Ubuntu/Debian/SUSE > Vendor PSIRT > GHSA/OSV > NVD; CERTs enrich; KEV sets flags.
|
||||
- Range selection uses comparers: NevraComparer, DebEvrComparer, SemVerRange; deterministic tie-breakers.
|
||||
- Provenance propagation merges unique entries; references deduped by (url, type).
|
||||
|
||||
## Configuration
|
||||
- Precedence overrides bind via `concelier:merge:precedence:ranks` (dictionary of `source` → `rank`, lower wins). Absent entries fall back to defaults.
|
||||
- Operator workflow: update `etc/concelier.yaml` or environment variables, restart merge job; overrides surface in metrics/logs as `AdvisoryOverride` entries.
|
||||
## In/Out of scope
|
||||
In: merge logic, precedence policy, hashing, event records, comparers.
|
||||
Out: fetching/parsing, exporter packaging, signing.
|
||||
## Observability & security expectations
|
||||
- Metrics: merge.delta.count, merge.identity.conflicts, merge.range.overrides, merge.duration_ms.
|
||||
- Logs: decisions (why replaced), keys involved, hashes; avoid dumping large blobs; redact secrets (none expected).
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Merge.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -1,30 +1,41 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Canonical data model for normalized advisories and all downstream serialization. Source of truth for merge/export.
|
||||
## Scope
|
||||
- Canonical types: Advisory, AdvisoryReference, CvssMetric, AffectedPackage, AffectedVersionRange, AdvisoryProvenance.
|
||||
- Invariants: stable ordering, culture-invariant serialization, UTC timestamps, deterministic equality semantics.
|
||||
- Field semantics: preserve all aliases/references; ranges per ecosystem (NEVRA/EVR/SemVer); provenance on every mapped field.
|
||||
- Backward/forward compatibility: additive evolution; versioned DTOs where needed; no breaking field renames.
|
||||
- Detailed field coverage documented in `CANONICAL_RECORDS.md`; update alongside model changes.
|
||||
## Participants
|
||||
- Source connectors map external DTOs into these types.
|
||||
- Merge engine composes/overrides AffectedPackage sets and consolidates references/aliases.
|
||||
- Exporters serialize canonical documents deterministically.
|
||||
## Interfaces & contracts
|
||||
- Null-object statics: Advisory.Empty, AdvisoryReference.Empty, CvssMetric.Empty.
|
||||
- AffectedPackage.Type describes semantics (e.g., rpm, deb, cpe, semver). Identifier is stable (e.g., NEVRA, PURL, CPE).
|
||||
- Version ranges list is ordered by introduction then fix; provenance identifies source/kind/value/recordedAt.
|
||||
- Alias schemes must include CVE, GHSA, OSV, JVN/JVNDB, BDU, VU(CERT/CC), MSRC, CISCO-SA, ORACLE-CPU, APSB/APA, APPLE-HT, CHROMIUM-POST, VMSA, RHSA, USN, DSA, SUSE-SU, ICSA, CWE, CPE, PURL.
|
||||
## In/Out of scope
|
||||
In: data shapes, invariants, helpers for canonical serialization and comparison.
|
||||
Out: fetching/parsing external schemas, storage, HTTP.
|
||||
## Observability & security expectations
|
||||
- No secrets; purely in-memory types.
|
||||
- Provide debug renders for test snapshots (canonical JSON).
|
||||
- Emit model version identifiers in logs when canonical structures change; keep adapters for older readers until deprecated.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Models.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
# AGENTS
|
||||
## Role
|
||||
Canonical data model for normalized advisories and all downstream serialization. Source of truth for merge/export.
|
||||
## Scope
|
||||
- Canonical types: Advisory, AdvisoryReference, CvssMetric, AffectedPackage, AffectedVersionRange, AdvisoryProvenance.
|
||||
- Invariants: stable ordering, culture-invariant serialization, UTC timestamps, deterministic equality semantics.
|
||||
- Field semantics: preserve all aliases/references; ranges per ecosystem (NEVRA/EVR/SemVer); provenance on every mapped field.
|
||||
- Backward/forward compatibility: additive evolution; versioned DTOs where needed; no breaking field renames.
|
||||
- Detailed field coverage documented in `CANONICAL_RECORDS.md`; update alongside model changes.
|
||||
## Participants
|
||||
- Source connectors map external DTOs into these types.
|
||||
- Merge engine composes/overrides AffectedPackage sets and consolidates references/aliases.
|
||||
- Exporters serialize canonical documents deterministically.
|
||||
## Interfaces & contracts
|
||||
- Null-object statics: Advisory.Empty, AdvisoryReference.Empty, CvssMetric.Empty.
|
||||
- AffectedPackage.Type describes semantics (e.g., rpm, deb, cpe, semver). Identifier is stable (e.g., NEVRA, PURL, CPE).
|
||||
- Version ranges list is ordered by introduction then fix; provenance identifies source/kind/value/recordedAt.
|
||||
- Alias schemes must include CVE, GHSA, OSV, JVN/JVNDB, BDU, VU(CERT/CC), MSRC, CISCO-SA, ORACLE-CPU, APSB/APA, APPLE-HT, CHROMIUM-POST, VMSA, RHSA, USN, DSA, SUSE-SU, ICSA, CWE, CPE, PURL.
|
||||
## In/Out of scope
|
||||
In: data shapes, invariants, helpers for canonical serialization and comparison.
|
||||
Out: fetching/parsing external schemas, storage, HTTP.
|
||||
## Observability & security expectations
|
||||
- No secrets; purely in-memory types.
|
||||
- Provide debug renders for test snapshots (canonical JSON).
|
||||
- Emit model version identifiers in logs when canonical structures change; keep adapters for older readers until deprecated.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Models.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
# Concelier Normalization Guild Charter
|
||||
|
||||
## Mission
|
||||
Maintain helper utilities that normalise upstream advisory payloads into Concelier’s immutable observation/linkset structures while respecting the Aggregation-Only Contract. The goal is to provide consistent field extraction and canonicalisation without introducing derived data or losing provenance.
|
||||
|
||||
## Scope
|
||||
- Shared normalization helpers in `StellaOps.Concelier.Normalization`.
|
||||
- Field mappers (identifiers, products, references) reused by connectors.
|
||||
- Canonical JSON serialization rules (sorting, formatting, timestamp normalisation).
|
||||
- Schema evolution helpers tied to `advisory_raw` / `linkset` documents.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/ingestion/aggregation-only-contract.md`
|
||||
- `docs/modules/concelier/operations/conflict-resolution.md`
|
||||
- `docs/modules/concelier/operations/connectors/*` (to understand per-source nuances)
|
||||
- `docs/modules/concelier/design/` materials if referenced by tasks.
|
||||
|
||||
## Working Agreement
|
||||
1. **Synchronise status** in both `docs/implplan/SPRINTS.md` and `TASKS.md` when starting/finishing tasks.
|
||||
2. **AOC compliance**: avoid adding severity, consensus, fix hints, or other derived fields—output raw upstream data plus provenance.
|
||||
3. **Deterministic outputs**: enforce stable ordering (sorted arrays/objects), UTC timestamps, lowercase enum values as documented.
|
||||
4. **Shared API stability**: version helpers when breaking changes are needed; communicate with connector guilds.
|
||||
5. **Testing**: extend golden fixtures & property tests to catch regressions; ensure CI covers multi-source scenarios.
|
||||
6. **Documentation**: update developer notes (add/refresh doc under `docs/modules/concelier`) when normalization contracts change.
|
||||
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
@@ -1,29 +1,40 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Canonical persistence for raw documents, DTOs, canonical advisories, jobs, and state. Provides repositories and bootstrapper for collections/indexes.
|
||||
## Scope
|
||||
- Collections (MongoStorageDefaults): source, source_state, document, dto, advisory, alias, affected, reference, kev_flag, ru_flags, jp_flags, psirt_flags, merge_event, export_state, locks, jobs; GridFS bucket fs.documents; field names include ttlAt (locks), sourceName, uri, advisoryKey.
|
||||
- Records: SourceState (cursor, lastSuccess/error, failCount, backoffUntil), JobRun, MergeEvent, ExportState, Advisory documents mirroring Models with embedded arrays when practical.
|
||||
- Bootstrapper: create collections, indexes (unique advisoryKey, scheme/value, platform/name, published, modified), TTL on locks, and validate connectivity for /ready health probes.
|
||||
- Job store: create, read, mark completed/failed; compute durations; recent/last queries; active by status.
|
||||
- Advisory store: CRUD for canonical advisories; query by key/alias and list for exporters with deterministic paging.
|
||||
## Participants
|
||||
- Core jobs read/write runs and leases; WebService /ready pings database; /jobs APIs query runs/definitions.
|
||||
- Source connectors store raw docs, DTOs, and mapped canonical advisories with provenance; Update SourceState cursor/backoff.
|
||||
- Exporters read advisories and write export_state.
|
||||
## Interfaces & contracts
|
||||
- IMongoDatabase injected; MongoUrl from options; database name from options or MongoUrl or default "concelier".
|
||||
- Repositories expose async methods with CancellationToken; deterministic sorting.
|
||||
- All date/time values stored as UTC; identifiers normalized.
|
||||
## In/Out of scope
|
||||
In: persistence, bootstrap, indexes, basic query helpers.
|
||||
Out: business mapping logic, HTTP, packaging.
|
||||
## Observability & security expectations
|
||||
- Log collection/index creation; warn on existing mismatches.
|
||||
- Timeouts and retry policies; avoid unbounded scans; page reads.
|
||||
- Do not log DSNs with credentials; redact in diagnostics.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Storage.Mongo.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
# AGENTS
|
||||
## Role
|
||||
Canonical persistence for raw documents, DTOs, canonical advisories, jobs, and state. Provides repositories and bootstrapper for collections/indexes.
|
||||
## Scope
|
||||
- Collections (MongoStorageDefaults): source, source_state, document, dto, advisory, alias, affected, reference, kev_flag, ru_flags, jp_flags, psirt_flags, merge_event, export_state, locks, jobs; GridFS bucket fs.documents; field names include ttlAt (locks), sourceName, uri, advisoryKey.
|
||||
- Records: SourceState (cursor, lastSuccess/error, failCount, backoffUntil), JobRun, MergeEvent, ExportState, Advisory documents mirroring Models with embedded arrays when practical.
|
||||
- Bootstrapper: create collections, indexes (unique advisoryKey, scheme/value, platform/name, published, modified), TTL on locks, and validate connectivity for /ready health probes.
|
||||
- Job store: create, read, mark completed/failed; compute durations; recent/last queries; active by status.
|
||||
- Advisory store: CRUD for canonical advisories; query by key/alias and list for exporters with deterministic paging.
|
||||
## Participants
|
||||
- Core jobs read/write runs and leases; WebService /ready pings database; /jobs APIs query runs/definitions.
|
||||
- Source connectors store raw docs, DTOs, and mapped canonical advisories with provenance; Update SourceState cursor/backoff.
|
||||
- Exporters read advisories and write export_state.
|
||||
## Interfaces & contracts
|
||||
- IMongoDatabase injected; MongoUrl from options; database name from options or MongoUrl or default "concelier".
|
||||
- Repositories expose async methods with CancellationToken; deterministic sorting.
|
||||
- All date/time values stored as UTC; identifiers normalized.
|
||||
## In/Out of scope
|
||||
In: persistence, bootstrap, indexes, basic query helpers.
|
||||
Out: business mapping logic, HTTP, packaging.
|
||||
## Observability & security expectations
|
||||
- Log collection/index creation; warn on existing mismatches.
|
||||
- Timeouts and retry policies; avoid unbounded scans; page reads.
|
||||
- Do not log DSNs with credentials; redact in diagnostics.
|
||||
## Tests
|
||||
- Author and review coverage in `../StellaOps.Concelier.Storage.Mongo.Tests`.
|
||||
- Shared fixtures (e.g., `MongoIntegrationFixture`, `ConnectorTestHarness`) live in `../StellaOps.Concelier.Testing`.
|
||||
- Keep fixtures deterministic; match new cases to real-world advisories or regression scenarios.
|
||||
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
|
||||
Reference in New Issue
Block a user