Add unit tests and implementations for MongoDB index models and OpenAPI metadata

- Implemented `MongoIndexModelTests` to verify index models for various stores. - Created `OpenApiMetadataFactory` with methods to generate OpenAPI metadata. - Added tests for `OpenApiMetadataFactory` to ensure expected defaults and URL overrides. - Introduced `ObserverSurfaceSecrets` and `WebhookSurfaceSecrets` for managing secrets. - Developed `RuntimeSurfaceFsClient` and `WebhookSurfaceFsClient` for manifest retrieval. - Added dependency injection tests for `SurfaceEnvironmentRegistration` in both Observer and Webhook contexts. - Implemented tests for secret resolution in `ObserverSurfaceSecretsTests` and `WebhookSurfaceSecretsTests`. - Created `EnsureLinkNotMergeCollectionsMigrationTests` to validate MongoDB migration logic. - Added project files for MongoDB tests and NuGet package mirroring.
2025-11-17 21:21:56 +02:00
parent d3128aec24
commit 9075bad2d9
146 changed files with 152183 additions and 82 deletions
--- a/docs/modules/concelier/link-not-merge-schema.md
+++ b/docs/modules/concelier/link-not-merge-schema.md
@@ -0,0 +1,125 @@
+# Link-Not-Merge (LNM) Observation & Linkset Schema
+
+_Draft for approval — authored 2025-11-16 to unblock CONCELIER-LNM tracks._
+
+## Goals
+- Immutable storage of raw advisory observations per source/tenant.
+- Deterministic linksets built from observations without merging or mutating originals.
+- Stable across online/offline deployments; replayable from raw inputs.
+
+## Observation document (Mongo JSON Schema excerpt)
+```json
+{
+  "bsonType": "object",
+  "required": ["_id","tenantId","source","advisoryId","affected","provenance","ingestedAt"],
+  "properties": {
+    "_id": {"bsonType": "objectId"},
+    "tenantId": {"bsonType": "string"},
+    "source": {"bsonType": "string", "description": "Adapter id, e.g., ghsa, nvd, cert-bund"},
+    "advisoryId": {"bsonType": "string"},
+    "title": {"bsonType": "string"},
+    "summary": {"bsonType": "string"},
+    "severities": {
+      "bsonType": "array",
+      "items": {"bsonType": "object", "required": ["system","score"],
+        "properties": {"system":{"bsonType":"string"},"score":{"bsonType":"double"},"vector":{"bsonType":"string"}}}
+    },
+    "affected": {
+      "bsonType": "array",
+      "items": {"bsonType":"object","required":["purl"],
+        "properties": {
+          "purl": {"bsonType":"string"},
+          "package": {"bsonType":"string"},
+          "versions": {"bsonType":"array","items":{"bsonType":"string"}},
+          "ranges": {"bsonType":"array","items":{"bsonType":"object",
+            "required":["type","events"],
+            "properties": {"type":{"bsonType":"string"},"events":{"bsonType":"array","items":{"bsonType":"object"}}}}},
+          "ecosystem": {"bsonType":"string"},
+          "cpe": {"bsonType":"array","items":{"bsonType":"string"}},
+          "cpes": {"bsonType":"array","items":{"bsonType":"string"}}
+        }
+      }
+    },
+    "references": {"bsonType": "array", "items": {"bsonType":"string"}},
+    "weaknesses": {"bsonType":"array","items":{"bsonType":"string"}},
+    "published": {"bsonType": "date"},
+    "modified": {"bsonType": "date"},
+    "provenance": {
+      "bsonType": "object",
+      "required": ["sourceArtifactSha","fetchedAt"],
+      "properties": {
+        "sourceArtifactSha": {"bsonType":"string"},
+        "fetchedAt": {"bsonType":"date"},
+        "ingestJobId": {"bsonType":"string"},
+        "signature": {"bsonType":"object"}
+      }
+    },
+    "ingestedAt": {"bsonType": "date"}
+  }
+}
+```
+
+### Observation invariants
+- **Immutable:** no in-place updates; new revision → new document with `supersedesId` optional pointer.
+- **Deterministic keying:** `_id` derived from `hash(tenantId|source|advisoryId|provenance.sourceArtifactSha)` to keep inserts idempotent in replay.
+- **Normalization guardrails:** version ranges must be stored as raw-from-source; no inferred merges.
+
+## Linkset document
+```json
+{
+  "bsonType":"object",
+  "required":["_id","tenantId","advisoryId","source","observations","createdAt"],
+  "properties":{
+    "_id":{"bsonType":"objectId"},
+    "tenantId":{"bsonType":"string"},
+    "advisoryId":{"bsonType":"string"},
+    "source":{"bsonType":"string"},
+    "observations":{"bsonType":"array","items":{"bsonType":"objectId"}},
+    "normalized": {
+      "bsonType":"object",
+      "properties":{
+        "purls":{"bsonType":"array","items":{"bsonType":"string"}},
+        "versions":{"bsonType":"array","items":{"bsonType":"string"}},
+        "ranges": {"bsonType":"array","items":{"bsonType":"object"}},
+        "severities": {"bsonType":"array","items":{"bsonType":"object"}}
+      }
+    },
+    "createdAt":{"bsonType":"date"},
+    "builtByJobId":{"bsonType":"string"},
+    "provenance": {"bsonType":"object","properties":{
+      "observationHashes":{"bsonType":"array","items":{"bsonType":"string"}},
+      "toolVersion" : {"bsonType":"string"},
+      "policyHash" : {"bsonType":"string"}
+    }}
+  }
+}
+```
+
+### Linkset invariants
+- Built from a set of observation IDs; never overwrites observations.
+- Carries the hash list of source observations for audit/replay.
+- Deterministic sort: observations sorted by `source, advisoryId, fetchedAt` before hashing.
+
+## Indexes (Mongo)
+- Observations: `{ tenantId:1, source:1, advisoryId:1, provenance.fetchedAt:-1 }` (compound for ingest); `{ provenance.sourceArtifactSha:1 }` unique to avoid dup writes.
+- Linksets: `{ tenantId:1, advisoryId:1, source:1 }` unique; `{ observations:1 }` sparse for reverse lookups.
+
+## Collections
+- `advisory_observations` — raw per-source docs (immutable).
+- `advisory_linksets` — derived normalized aggregates with observation pointers and hashes.
+
+## Determinism & replay
+- Replay rebuild: order observations by fetchedAt, recompute linkset hash list, ensure byte-identical linkset JSON.
+- All timestamps UTC ISO-8601; no server-local time.
+- String normalization: lowercase `source`, trim/normalize PURLs, stable sort arrays.
+
+## Sample documents
+See `docs/samples/lnm/observation-ghsa.json` and `docs/samples/lnm/linkset-ghsa.json` (added with this draft) for concrete payloads.
+
+## Approval path
+1) Architecture + Concelier Core review this document.
+2) If accepted, freeze JSON Schema and roll into `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo` migrations.
+3) Update consumers (policy/CLI/export) to read from linksets only; deprecate Merge endpoints.
+
+---
+Tracking: CONCELIER-LNM-21-001/002/101; Sprint 110 blockers (Concelier/Excititor waves).