feat: Add new projects to solution and implement contract testing documentation

- Added "StellaOps.Policy.Engine", "StellaOps.Cartographer", and "StellaOps.SbomService" projects to the StellaOps solution. - Created AGENTS.md to outline the Contract Testing Guild Charter, detailing mission, scope, and definition of done. - Established TASKS.md for the Contract Testing Task Board, outlining tasks for Sprint 62 and Sprint 63 related to mock servers and replay testing.
2025-10-27 07:57:55 +02:00
parent 1e41ba7ffa
commit 651b8e0fa3
355 changed files with 17276 additions and 1160 deletions
--- a/src/StellaOps.Concelier.Core/TASKS.md
+++ b/src/StellaOps.Concelier.Core/TASKS.md
@@ -1,21 +1,106 @@
-# TASKS
-| Task | Owner(s) | Depends on | Notes |
-|---|---|---|---|
-|JobCoordinator implementation (create/get/mark status)|BE-Core|Storage.Mongo|DONE – `JobCoordinator` drives Mongo-backed runs.|
-|Cron scheduling loop with TimeProvider|BE-Core|Core|DONE – `JobSchedulerHostedService` evaluates cron expressions.|
-|Single-flight/lease semantics|BE-Core|Storage.Mongo|DONE – lease acquisition backed by `MongoLeaseStore`.|
-|Trigger API contract (Result mapping)|BE-Core|WebService|DONE – `JobTriggerResult` outcomes map to HTTP statuses.|
-|Run telemetry enrichment|BE-Core|Observability|DONE – `JobDiagnostics` ties activities & counters into coordinator/scheduler paths.|
-|Deterministic params hashing|BE-Core|Core|DONE – `JobParametersHasher` creates SHA256 hash.|
-|Golden tests for timeout/cancel|QA|Core|DONE – JobCoordinatorTests cover cancellation timeout path.|
-|JobSchedulerBuilder options registry coverage|BE-Core|Core|DONE – added scheduler tests confirming cron/timeout/lease metadata persists via JobSchedulerOptions.|
-|Plugin discovery + DI glue with PluginHost|BE-Core|Plugin libs|DONE – JobPluginRegistrationExtensions now loads PluginHost routines and wires connector/exporter registrations.|
-|Harden lease release error handling in JobCoordinator|BE-Core|Storage.Mongo|DONE – lease release failures now logged, wrapped, and drive run failure status; fire-and-forget execution guarded. Verified with `dotnet test --no-build --filter JobCoordinator`.|
-|Validate job trigger parameters for serialization|BE-Core|WebService|DONE – trigger parameters normalized/serialized with defensive checks returning InvalidParameters on failure. Full-suite `dotnet test --no-build` currently red from live connector fixture drift (Oracle/JVN/RedHat).|
-|FEEDCORE-ENGINE-03-001 Canonical merger implementation|BE-Core|Merge|DONE – `CanonicalMerger` applies GHSA/NVD/OSV conflict rules with deterministic provenance and comprehensive unit coverage. **Coordination:** Connector leads must align mapper outputs with the canonical field expectations before 2025-10-18 so Merge can activate the path globally.|
-|FEEDCORE-ENGINE-03-002 Field precedence and tie-breaker map|BE-Core|Merge|DONE – field precedence and freshness overrides enforced via `FieldPrecedence` map with tie-breakers and analytics capture. **Reminder:** Storage/Merge owners review precedence overrides when onboarding new feeds to ensure `decisionReason` tagging stays consistent.|
-|Canonical merger parity for description/CWE/canonical metric|BE-Core|Models|DONE (2025-10-15) – merger now populates description/CWEs/canonical metric id with provenance and regression tests cover the new decisions.|
-|Reference normalization & freshness instrumentation cleanup|BE-Core, QA|Models|DONE (2025-10-15) – reference keys normalized, freshness overrides applied to union fields, and new tests assert decision logging.|
-|FEEDCORE-ENGINE-07-001 – Advisory event log & asOf queries|Team Core Engine & Storage Analytics|FEEDSTORAGE-DATA-07-001|**DONE (2025-10-19)** – Implemented `AdvisoryEventLog` service plus repository contracts, canonical hashing, and lower-cased key normalization with replay support; documented determinism guarantees. Tests: `dotnet test src/StellaOps.Concelier.Core.Tests/StellaOps.Concelier.Core.Tests.csproj`.|
-|FEEDCORE-ENGINE-07-002 – Noise prior computation service|Team Core Engine & Data Science|FEEDCORE-ENGINE-07-001|**DONE (2025-10-21)** – Build rule-based learner capturing false-positive priors per package/env, persist summaries, and expose APIs for Excititor/scan suppressors with reproducible statistics.|
-|FEEDCORE-ENGINE-07-003 – Unknown state ledger & confidence seeding|Team Core Engine & Storage Analytics|FEEDCORE-ENGINE-07-001|DONE (2025-10-21) – Persisted `unknown_vuln_range/unknown_origin/ambiguous_fix` markers with seeded confidence bands, exposed query surface for Policy, and added canonical serialization fixtures + regression tests.|
+# TASKS — Epic 1: Aggregation-Only Contract
+> **AOC Reminder:** ingestion aggregates and links only—no precedence, normalization, or severity computation. Derived data lives in Policy/overlay services.
+| ID | Status | Owner(s) | Depends on | Notes |
+|---|---|---|---|---|
+| CONCELIER-CORE-AOC-19-001 `AOC write guard` | TODO | Concelier Core Guild | WEB-AOC-19-001 | Implement repository interceptor that inspects write payloads for forbidden AOC keys, validates provenance/signature presence, and maps violations to `ERR_AOC_00x`. |
+| CONCELIER-CORE-AOC-19-002 `Deterministic linkset extraction` | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-001 | Build canonical linkset mappers for CVE/GHSA/PURL/CPE/reference extraction from upstream raw payloads, ensuring reconciled-from metadata is tracked and deterministic. |
+| CONCELIER-CORE-AOC-19-003 `Idempotent append-only upsert` | TODO | Concelier Core Guild | CONCELIER-STORE-AOC-19-002 | Implement idempotent upsert path using `(vendor, upstreamId, contentHash, tenant)` key, emitting supersedes pointers for new revisions and preventing duplicate inserts. |
+| CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003 | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only. |
+
+## Policy Engine v2
+
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-POLICY-20-002 `Linkset enrichment for policy` | TODO | Concelier Core Guild, Policy Guild | CONCELIER-CORE-AOC-19-002, POLICY-ENGINE-20-001 | Strengthen linkset builders with vendor-specific equivalence tables, NEVRA/PURL normalization, and version range parsing to maximize policy join recall; update fixtures + docs. |
+
+## Graph Explorer v1
+
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-GRAPH-21-001 `SBOM projection enrichment` | TODO | Concelier Core Guild, Cartographer Guild | CONCELIER-POLICY-20-002, CARTO-GRAPH-21-002 | Extend SBOM normalization to emit full relationship graph (depends_on/contains/provides), scope tags, entrypoint annotations, and component metadata required by Cartographer. |
+| CONCELIER-GRAPH-21-002 `Change events` | TODO | Concelier Core Guild, Scheduler Guild | CONCELIER-GRAPH-21-001 | Publish change events (new SBOM version, relationship delta) for Cartographer build queue; ensure events include tenant/context metadata. |
+
+## Link-Not-Merge v1
+
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-LNM-21-001 `Advisory observation schema` | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-001 | Introduce immutable `advisory_observations` model with AOC metadata, raw payload pointers, normalized fields, and tenancy guardrails; publish schema definition. |
+| CONCELIER-LNM-21-002 `Linkset builder` | TODO | Concelier Core Guild, Data Science Guild | CONCELIER-LNM-21-001 | Implement correlation pipeline (alias graph, PURL overlap, CVSS vector equality, fuzzy title match) that produces `advisory_linksets` with confidence + conflict annotations. |
+| CONCELIER-LNM-21-003 `Conflict annotator` | TODO | Concelier Core Guild | CONCELIER-LNM-21-002 | Detect field disagreements (severity, CVSS, ranges, references) and record structured conflicts on linksets; surface to API/UI. |
+| CONCELIER-LNM-21-004 `Merge code removal` | TODO | Concelier Core Guild | CONCELIER-LNM-21-002 | Excise existing merge/dedup logic, enforce immutability on observations, and add guards/tests to prevent future merges. |
+| CONCELIER-LNM-21-005 `Event emission` | TODO | Concelier Core Guild, Platform Events Guild | CONCELIER-LNM-21-002 | Emit `advisory.linkset.updated` events with delta payloads for downstream Policy Engine/Cartographer consumers; ensure idempotent delivery. |
+
+## Policy Engine + Editor v1
+
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-POLICY-23-001 `Evidence indexes` | TODO | Concelier Core Guild | CONCELIER-LNM-21-002 | Add secondary indexes/materialized views to accelerate policy lookups (alias, severity per observation, correlation confidence). Document query contracts for runtime. |
+| CONCELIER-POLICY-23-002 `Event guarantees` | TODO | Concelier Core Guild, Platform Events Guild | CONCELIER-LNM-21-005 | Ensure `advisory.linkset.updated` emits at-least-once with idempotent keys and include policy-relevant metadata (confidence, conflict summary). |
+
+## Graph & Vuln Explorer v1
+
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-GRAPH-24-001 `Advisory overlay inputs` | TODO | Concelier Core Guild | CONCELIER-POLICY-23-001 | Expose raw advisory observations/linksets with tenant filters for overlay services; no derived counts/severity in ingestion. |
+
+## Reachability v1
+
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-SIG-26-001 `Vulnerable symbol exposure` | TODO | Concelier Core Guild, Signals Guild | SIGNALS-24-002 | Expose advisory metadata (affected symbols/functions) via API to enrich reachability scoring; update fixtures. |
+
+## Orchestrator Dashboard
+
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-ORCH-32-001 `Source registry integration` | TODO | Concelier Core Guild | ORCH-SVC-32-001, AUTH-ORCH-32-001 | Register Concelier data sources with orchestrator (metadata, schedules, rate policies) and wire provenance IDs/security scopes. |
+| CONCELIER-ORCH-32-002 `Worker SDK adoption` | TODO | Concelier Core Guild | CONCELIER-ORCH-32-001, WORKER-GO-32-001, WORKER-PY-32-001 | Embed orchestrator worker SDK in ingestion loops, emit heartbeats/progress/artifact hashes, and enforce idempotency keys. |
+| CONCELIER-ORCH-33-001 `Control hook compliance` | TODO | Concelier Core Guild | CONCELIER-ORCH-32-002, ORCH-SVC-33-001, ORCH-SVC-33-002 | Honor orchestrator throttle/pause/retry actions, surface structured error classes, and persist safe checkpoints for resume. |
+| CONCELIER-ORCH-34-001 `Backfill + ledger linkage` | TODO | Concelier Core Guild | CONCELIER-ORCH-33-001, ORCH-SVC-33-003, ORCH-SVC-34-001 | Execute orchestrator-driven backfills, reuse artifact hashes to avoid duplicates, and link provenance to run ledger exports. |
+
+## Authority-Backed Scopes & Tenancy (Epic 14)
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-TEN-48-001 `Tenant-aware linking` | TODO | Concelier Core Guild | AUTH-TEN-47-001 | Ensure advisory normalization/linking runs per tenant with RLS enforcing isolation; emit capability endpoint reporting `merge=false`; update events with tenant context. |
+
+## Observability & Forensics (Epic 15)
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-OBS-50-001 `Telemetry adoption` | TODO | Concelier Core Guild, Observability Guild | TELEMETRY-OBS-50-001, TELEMETRY-OBS-50-002 | Replace ad-hoc logging with telemetry core across ingestion/linking pipelines; ensure spans/logs include tenant, source vendor, upstream id, content hash, and trace IDs. |
+| CONCELIER-OBS-51-001 `Metrics & SLOs` | TODO | Concelier Core Guild, DevOps Guild | CONCELIER-OBS-50-001, TELEMETRY-OBS-51-001 | Emit metrics for ingest latency (cold/warm), queue depth, aoc violation rate, and publish SLO burn-rate alerts (ingest P95 <30s cold / <5s warm). Ship dashboards + alert configs. |
+| CONCELIER-OBS-52-001 `Timeline events` | TODO | Concelier Core Guild | CONCELIER-OBS-50-001, TIMELINE-OBS-52-002 | Emit `timeline_event` records for advisory ingest/normalization/linkset creation with provenance, trace IDs, conflict summaries, and evidence placeholders. |
+| CONCELIER-OBS-53-001 `Evidence snapshots` | TODO | Concelier Core Guild, Evidence Locker Guild | CONCELIER-OBS-52-001, EVID-OBS-53-002 | Produce advisory evaluation bundle payloads (raw doc, linkset, normalization diff) for evidence locker; ensure Merkle manifests seeded with content hashes. |
+| CONCELIER-OBS-54-001 `Attestation & verification` | TODO | Concelier Core Guild, Provenance Guild | CONCELIER-OBS-53-001, PROV-OBS-54-001 | Attach DSSE attestations for advisory processing batches, expose verification API to confirm bundle integrity, and link attestation IDs back to timeline + ledger. |
+| CONCELIER-OBS-55-001 `Incident mode hooks` | TODO | Concelier Core Guild, DevOps Guild | CONCELIER-OBS-51-001, DEVOPS-OBS-55-001 | Increase sampling, capture raw payload snapshots, and extend retention under incident mode; emit activation events + guardrails against PII leak. |
+
+## Air-Gapped Mode (Epic 16)
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-AIRGAP-56-001 `Mirror ingestion adapters` | TODO | Concelier Core Guild | AIRGAP-IMP-57-002, MIRROR-CRT-56-001 | Add mirror source adapters reading advisories from imported bundles, preserving source metadata and bundle IDs. Ensure ingestion remains append-only. |
+| CONCELIER-AIRGAP-56-002 `Bundle catalog linking` | TODO | Concelier Core Guild, AirGap Importer Guild | CONCELIER-AIRGAP-56-001, AIRGAP-IMP-57-001 | Persist `bundle_id`, `merkle_root`, and time anchor references on observations/linksets for provenance. |
+| CONCELIER-AIRGAP-57-001 `Sealed-mode source restrictions` | TODO | Concelier Core Guild, AirGap Policy Guild | CONCELIER-AIRGAP-56-001, AIRGAP-POL-56-001 | Enforce sealed-mode egress rules by disallowing non-mirror connectors and surfacing remediation errors. |
+| CONCELIER-AIRGAP-57-002 `Staleness annotations` | TODO | Concelier Core Guild, AirGap Time Guild | CONCELIER-AIRGAP-56-002, AIRGAP-TIME-58-001 | Compute staleness metadata for advisories per bundle and expose via API for Console/CLI badges. |
+| CONCELIER-AIRGAP-58-001 `Portable advisory evidence` | TODO | Concelier Core Guild, Evidence Locker Guild | CONCELIER-OBS-53-001, EVID-OBS-54-001 | Package advisory evidence fragments into portable evidence bundles for cross-domain transfer. |
+
+## SDKs & OpenAPI (Epic 17)
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-OAS-61-001 `Spec coverage` | TODO | Concelier Core Guild, API Contracts Guild | OAS-61-001 | Update Concelier OAS with advisory observation/linkset endpoints, standard pagination, and source provenance fields. |
+| CONCELIER-OAS-61-002 `Examples library` | TODO | Concelier Core Guild | CONCELIER-OAS-61-001 | Provide rich examples for advisories, linksets, conflict annotations used by SDK + docs. |
+| CONCELIER-OAS-62-001 `SDK smoke tests` | TODO | Concelier Core Guild, SDK Generator Guild | CONCELIER-OAS-61-001, SDKGEN-63-001 | Add SDK tests covering advisory search, pagination, and conflict handling; ensure source metadata surfaced. |
+| CONCELIER-OAS-63-001 `Deprecation headers` | TODO | Concelier Core Guild, API Governance Guild | APIGOV-63-001 | Implement deprecation header support and timeline events for retiring endpoints. |
+
+## Risk Profiles (Epic 18)
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-RISK-66-001 `CVSS/KEV providers` | TODO | Concelier Core Guild, Risk Engine Guild | RISK-ENGINE-67-001 | Expose CVSS, KEV, fix availability data via provider APIs with source metadata preserved. |
+| CONCELIER-RISK-66-002 `Fix availability signals` | TODO | Concelier Core Guild | CONCELIER-RISK-66-001 | Provide structured fix availability and release metadata consumable by risk engine; document provenance. |
+| CONCELIER-RISK-67-001 `Source consensus metrics` | TODO | Concelier Core Guild | CONCELIER-RISK-66-001 | Add consensus counts and confidence scores for linked advisories; ensure explainability includes source digests. |
+| CONCELIER-RISK-68-001 `Policy Studio integration` | TODO | Concelier Core Guild, Policy Studio Guild | POLICY-RISK-68-001 | Surface advisory fields in Policy Studio profile editor (signal pickers, reducers). |
+| CONCELIER-RISK-69-001 `Notification hooks` | TODO | Concelier Core Guild, Notifications Guild | CONCELIER-RISK-66-002 | Emit events when advisory signals change impacting risk scores (e.g., fix available). |
+
+## Attestor Console (Epic 19)
+| ID | Status | Owner(s) | Depends on | Notes |
+|----|--------|----------|------------|-------|
+| CONCELIER-ATTEST-73-001 `ScanResults attestation inputs` | TODO | Concelier Core Guild, Attestor Service Guild | ATTEST-TYPES-72-001 | Provide normalized advisory data and linkset digests needed for ScanResults attestations. |
+| CONCELIER-ATTEST-73-002 `Transparency metadata` | TODO | Concelier Core Guild | CONCELIER-ATTEST-73-001 | Ensure Conseiller exposes source digests for transparency proofs and explainability. |