Merge branch 'main' of https://git.stella-ops.org/stella-ops.org/git.stella-ops.org

2025-10-30 00:12:41 +02:00
parent e8537460a3 fac626db8d
commit 0bc882e75a
39 changed files with 2130 additions and 164 deletions
--- a/docs/dev/cartographer-graph-handshake.md
+++ b/docs/dev/cartographer-graph-handshake.md
@@ -0,0 +1,49 @@
+# Cartographer Graph Handshake Plan
+
+_Status: 2025-10-29_
+
+## Why this exists
+The Concelier/Excititor graph enrichment work (CONCELIER-GRAPH-21-001/002, EXCITITOR-GRAPH-21-001/002/005) and the merge-side coordination tasks (FEEDMERGE-COORD-02-901/902) are blocked on a clear contract with Cartographer and the Policy Engine. This document captures the minimum artefacts each guild owes so we can unblock the graph pipeline and resume implementation without re-scoping every stand-up.
+
+## Deliverables by guild
+
+### Cartographer Guild
+- **CARTO-GRAPH-21-002** (Inspector contract): publish the inspector payload schema (`graph.inspect.v1`) including the fields Cartographer needs from Concelier/Excititor (SBOM relationships, advisory/VEX linkouts, justification summaries). Target format: shared Proto/JSON schema stored under `src/Cartographer/Contracts/`.
+- **CARTO-GRAPH-21-005** (Inspector access patterns): document the query shapes Cartographer will execute (PURL → advisory, PURL → VEX statement, policy scope filters) so storage can project the right indexes/materialized views. Include sample `mongosh` queries and desired TTL/limit behaviour.
+- Provide a test harness (e.g., Postman collection or integration fixture) Cartographer will use to validate the Concelier/Excititor endpoints once they land.
+
+### Concelier Core Guild
+- Derive adjacency data from SBOM normalization as described in CONCELIER-GRAPH-21-001 (depends on `CONCELIER-POLICY-20-002`). Once Cartographer publishes the schema above, implement:
+  - Node payloads: component metadata, scopes, entrypoint annotations.
+  - Edge payloads: `contains`, `depends_on`, `provides`, provenance array.
+- **Change events (CONCELIER-GRAPH-21-002)**: define `sbom.relationship.changed` event contract with tenant + context metadata, referencing Cartographer’s filter requirements. Include event samples and replay instructions in `docs/graph/concelier-events.md`.
+- Coordinate with Cartographer on pagination/streaming expectations (page size, continuation token, retention window).
+
+### Excititor Core & Storage Guilds
+- **Inspector linkouts (EXCITITOR-GRAPH-21-001)**: expose Batched VEX/advisory lookup endpoint that accepts graph node PURLs and responds with raw document slices + justification metadata. Ensure Policy Engine scope enrichment (EXCITITOR-POLICY-20-002) feeds this response so Cartographer does not need to call multiple services.
+- **Overlay enrichment (EXCITITOR-GRAPH-21-002)**: align the overlay metadata with Cartographer’s schema once it lands (include justification summaries, document versions, and provenance).
+- **Indexes/materialized views (EXCITITOR-GRAPH-21-005)**: after Cartographer publishes query shapes, create the necessary indexes (PURL + tenant, policy scope) and document migrations in storage runbooks. Provide load testing evidence before enabling in production.
+
+### Policy Guild
+- **CONCELIER-POLICY-20-002**: publish the enriched linkset schema that powers both Concelier and Excititor payloads. Include enumerations for relationship types and scope tags.
+- Share the Policy Engine timeline for policy overlay metadata (`POLICY-ENGINE-30-001`) so Excititor can plan the overlay enrichment delivery.
+
+## Shared action items
+
+| Owner | Task | Deadline | Notes |
+|-------|------|----------|-------|
+| Cartographer | Publish inspector schema + query patterns (`CARTO-GRAPH-21-002`/`21-005`) | 2025-11-04 | Attach schema files + examples to this doc once merged. |
+| Concelier Core | Draft change-event payload with sample JSON | 2025-11-06 | Blocked until Cartographer schema lands; prepare skeleton PR in `docs/graph/concelier-events.md`. |
+| Excititor Core/Storage | Prototype batch linkout API + index design doc | 2025-11-07 | Leverage Cartographer query patterns to size indexes; include perf targets. |
+| Policy Guild | Confirm linkset enrichment fields + overlay timeline | 2025-11-05 | Needed to unblock both Concelier enrichment and Excititor overlay tasks. |
+
+## Reporting
+- Track progress in the `#cartographer-handshake` Slack thread (create once Cartographer posts the schema MR).
+- During the twice-weekly graph sync, review outstanding checklist items above and update the task notes (`TASKS.md`) so the backlog reflects real-time status.
+- Once the schema and query contracts are merged, the Concelier/Excititor teams can flip their tasks from **BLOCKED** to **DOING** and attach implementation plans referencing this document.
+
+## Appendix: references
+- `CONCELIER-GRAPH-21-001`, `CONCELIER-GRAPH-21-002` (Concelier Core task board)
+- `EXCITITOR-GRAPH-21-001`, `EXCITITOR-GRAPH-21-002`, `EXCITITOR-GRAPH-21-005` (Excititor Core/Storage task boards)
+- `CARTO-GRAPH-21-002`, `CARTO-GRAPH-21-005` (Cartographer task board)
+- `POLICY-ENGINE-30-001`, `CONCELIER-POLICY-20-002`, `EXCITITOR-POLICY-20-002` (Policy Engine roadmap)
--- a/docs/dev/java-analyzer-observation-plan.md
+++ b/docs/dev/java-analyzer-observation-plan.md
@@ -0,0 +1,37 @@
+# Java Analyzer Observation Writer Plan
+
+_Status: 2025-10-29_
+
+SCANNER-ANALYZERS-JAVA-21-008 (resolver + AOC writer) is blocked by upstream heuristics that need to settle before we can emit observation JSON. This note itemises the remaining work so the analyzer guild can sequence delivery without re-opening design discussions in every stand-up.
+
+## Prerequisite summary
+- **SCANNER-ANALYZERS-JAVA-21-004** (reflection / dynamic loader heuristics) – must emit normalized reflection edges with confidence + call-site metadata. Outstanding items: TCCL coverage for servlet containers and resource-based plugin hints. Owners: Java Analyzer Guild.
+- **SCANNER-ANALYZERS-JAVA-21-005** (framework config extraction) – required to surface Spring/Jakarta entrypoints that feed observation entrypoint metadata. Add YAML/property parsing fixtures and document reason codes (`config-spring`, `config-jaxrs`, etc.).
+- **SCANNER-ANALYZERS-JAVA-21-006** (JNI/native hints) – optional but highly recommended before observation writer so JNI edges land alongside static ones. Coordinate with native analyzer on reason codes.
+- **Advisory core** – ensure AOC writer schema (`JavaObservation.json`) is frozen before we serialise to avoid churn downstream.
+
+## Deliverables for SCANNER-ANALYZERS-JAVA-21-008
+1. **Observation projection (`JavaObservationWriter`)**
+   - Inputs: normalised workspace + analyzer outputs (classpath graph, SPI table, reflection edges, config hints, JNI hints).
+   - Outputs: deterministic JSON containing entrypoints, components, edges, warnings, provenance. Align with `docs/aoc/java-observation-schema.md` once published.
+2. **AOC guard integration**
+   - Serialize observation documents through `Scanner.Aoc` guard pipeline; add unit tests covering required fields and forbidden derived data.
+3. **Fixture updates**
+   - Expand `fixtures/lang/java/` set to include reflection-heavy app, Spring Boot sample, JNI sample, modular app. Record golden outputs with `UPDATE_JAVA_FIXTURES=1`.
+4. **Metrics & logging**
+   - Emit counters (`scanner.java.observation.edges_total`, etc.) to trace observation completeness during CI runs.
+5. **Documentation**
+   - Update `docs/scanner/java-analyzer.md` with reason code matrix and observation field definitions.
+
+## Action items
+| Owner | Task | Due | Notes |
+|-------|------|-----|-------|
+| Java Analyzer Guild | Land reflection TODOs (TCCL + resource plugin hints) | 2025-11-01 | Required for reliable dynamic edges. |
+| Java Analyzer Guild | Finish config extractor for Spring/Jakarta | 2025-11-02 | Use sample apps in `fixtures/lang/java/config-*`. |
+| Java Analyzer Guild | Draft observation writer spike PR using new schema | 2025-11-04 | PR can be draft but should include JSON schema + sample. |
+| Scanner AOC Owners | Validate observation JSON against AOC guard + schema | 2025-11-05 | Blocker for marking 21-008 as DOING. |
+| QA Guild | Prepare regression harness + performance gate (<300 ms per fat jar) | 2025-11-06 | Align with SCANNER-ANALYZERS-JAVA-21-009. |
+
+## Reporting
+- Track these checkpoints in the Java analyzer weekly sync; once prerequisites are green, flip SCANNER-ANALYZERS-JAVA-21-008 to **DOING**.
+- Store schema and sample output under `docs/scanner/java-observations/` so AOC reviewers have a stable reference.
--- a/docs/dev/normalized-rule-recipes.md
+++ b/docs/dev/normalized-rule-recipes.md
@@ -0,0 +1,94 @@
+# Normalized Version Rule Recipes
+
+_Status: 2025-10-29_
+
+This guide captures the minimum wiring required for connectors and Merge coordination tasks to finish the normalized-version rollout that unblocks FEEDMERGE-COORD-02-9xx.
+
+## 1. Quick-start checklist
+
+1. Ensure your mapper already emits `AffectedPackage.VersionRanges` (SemVer, NEVRA, EVR).  If you only have vendor/product strings, capture the raw range text before trimming so it can feed the helper.
+2. Call `SemVerRangeRuleBuilder.BuildNormalizedRules(rawRange, patchedVersion, provenance)` for each range and place the result in `AffectedPackage.NormalizedVersions`.
+3. Set a provenance note in the format `connector:{advisoryId}:{index}` so Merge can differentiate connector-provided rules from canonical fallbacks.
+4. Verify with `dotnet test` that the connector snapshot fixtures now include the `normalizedVersions` array and update fixtures by setting the connector-specific `UPDATE_*_FIXTURES=1` environment variable.
+5. Tail Merge logs (or the test output) for the new warning `Normalized version rules missing for {AdvisoryKey}`; an empty warning stream means the connector/merge artefacts are ready to close FEEDMERGE-COORD-02-901/902.
+
+## 2. Code snippet: SemVer connector (CCCS/Cisco/ICS-CISA)
+
+```csharp
+using StellaOps.Concelier.Normalization.SemVer;
+
+private static IReadOnlyList<AffectedPackage> BuildPackages(MyDto dto, DateTimeOffset recordedAt)
+{
+    var packages = new List<AffectedPackage>();
+
+    foreach (var entry in dto.AffectedEntries.Select((value, index) => (value, index)))
+    {
+        var rangeText = entry.value.Range?.Trim();
+        var patched = entry.value.FixedVersion;
+        var provenance = $"{MyConnectorPlugin.SourceName}:{dto.AdvisoryId}:{entry.index}";
+
+        var normalizedRules = SemVerRangeRuleBuilder.BuildNormalizedRules(rangeText, patched, provenance);
+        var primitives = SemVerRangeRuleBuilder.Build(rangeText, patched, provenance)
+            .Select(result => result.Primitive.ToAffectedVersionRange(provenance))
+            .ToArray();
+
+        packages.Add(new AffectedPackage(
+            AffectedPackageTypes.SemVer,
+            entry.value.PackageId,
+            versionRanges: primitives,
+            normalizedVersions: normalizedRules,
+            provenance: new[]
+            {
+                new AdvisoryProvenance(
+                    MyConnectorPlugin.SourceName,
+                    "package",
+                    entry.value.PackageId,
+                    recordedAt,
+                    new[] { ProvenanceFieldMasks.AffectedPackages })
+            }));
+    }
+
+    return packages;
+}
+```
+
+A few notes:
+
+- If you already have `SemVerPrimitive` instances, call `.ToNormalizedVersionRule(provenance)` on each primitive instead of rebuilding from raw strings.
+- Use `SemVerRangeRuleBuilder.BuildNormalizedRules` when the connector only tracks raw range text plus an optional fixed/patched version.
+- For products that encode ranges like `"ExampleOS 4.12 - 4.14"`, run a small regex to peel off the version substring (see §3) and use the same provenance note when emitting the rule and the original range primitive.
+
+## 3. Parsing helper for trailing version phrases
+
+Many of the overdue connectors store affected products as natural-language phrases.  The following helper normalises common patterns (`1.2 - 1.4`, `<= 3.5`, `Version 7.2 and later`).
+
+```csharp
+private static string? TryExtractRangeSuffix(string productString)
+{
+    if (string.IsNullOrWhiteSpace(productString))
+    {
+        return null;
+    }
+
+    var match = Regex.Match(productString, "(?<range>(?:<=?|>=?)?\s*\d+(?:\.\d+){0,2}(?:\s*-\s*\d+(?:\.\d+){0,2})?)", RegexOptions.CultureInvariant);
+    return match.Success ? match.Groups["range"].Value.Trim() : null;
+}
+```
+
+Once you extract the `range` fragment, feed it to `SemVerRangeRuleBuilder.BuildNormalizedRules(range, null, provenance)`.  Keep the original product string as-is so operators can still see the descriptive text.
+
+## 4. Merge dashboard hygiene
+
+- Run `dotnet build src/Concelier/__Libraries/StellaOps.Concelier.Merge/StellaOps.Concelier.Merge.csproj` after wiring a connector to confirm no warnings appear.
+- Merge counter tag pairs to watch in Grafana/CI logs:
+  - `concelier.merge.normalized_rules{package_type="npm"}` – increases once the connector emits normalized arrays.
+  - `concelier.merge.normalized_rules_missing{package_type="vendor"}` – should trend to zero once rollout completes.
+- The Merge service now logs `Normalized version rules missing for {AdvisoryKey}; sources=...; packageTypes=...` when a connector still needs to supply normalized rules.  Use this as the acceptance gate for FEEDMERGE-COORD-02-901/902.
+
+## 5. Documentation touchpoints
+
+- Update the connector `TASKS.md` entry with the date you flipped on normalized rules and note the provenance format you chose.
+- Record any locale-specific parsing (e.g., German `bis`) in the connector README so future contributors can regenerate fixtures confidently.
+- When opening the PR, include `dotnet test` output covering the connector tests so reviewers see the normalized array diff.
+
+Once each connector follows the steps above, we can mark FEEDCONN-CCCS-02-009, FEEDCONN-CISCO-02-009, FEEDCONN-CERTBUND-02-010, FEEDCONN-ICSCISA-02-012, and the FEEDMERGE-COORD-02-90x tasks as resolved.