router planning

2025-12-02 18:38:32 +02:00
parent 790801f329
commit 0c9e8d5d18
15 changed files with 6439 additions and 0 deletions
--- a/docs/router/10-Step.md
+++ b/docs/router/10-Step.md
@@ -0,0 +1,586 @@
+For this step you’re wiring **configuration** into the system properly:
+
+* Router reads a strongly‑typed config model (including payload limits, node region, transports).
+* Microservices can optionally load a YAML file to **override** endpoint metadata discovered by reflection.
+* No behavior changes to routing or transports, just how they get their settings.
+
+Think “config plumbing and merging rules,” not new business logic.
+
+---
+
+## 0. Preconditions
+
+Before starting, confirm:
+
+* `__Libraries/StellaOps.Router.Config` project exists and references `StellaOps.Router.Common`.
+* `StellaOps.Microservice` has:
+
+  * `StellaMicroserviceOptions` (ServiceName, Version, Region, InstanceId, Routers, ConfigFilePath).
+  * Reflection‑based endpoint discovery that produces `EndpointDescriptor` instances.
+* Gateway and microservices currently use **hardcoded** or stub config; you’re about to replace that with real config.
+
+---
+
+## 1. Define RouterConfig model and YAML schema
+
+**Project:** `__Libraries/StellaOps.Router.Config`
+**Owner:** config / platform dev
+
+### 1.1 C# model
+
+Create clear, minimal models to cover current needs (you can extend later):
+
+```csharp
+namespace StellaOps.Router.Config;
+
+public sealed class RouterConfig
+{
+    public GatewayNodeConfig Node { get; set; } = new();
+    public PayloadLimits PayloadLimits { get; set; } = new();
+    public IList<TransportEndpointConfig> Transports { get; set; } = new List<TransportEndpointConfig>();
+    public IList<ServiceConfig> Services { get; set; } = new List<ServiceConfig>();
+}
+
+public sealed class GatewayNodeConfig
+{
+    public string NodeId { get; set; } = string.Empty;
+    public string Region { get; set; } = string.Empty;
+    public string Environment { get; set; } = "prod";
+}
+
+public sealed class TransportEndpointConfig
+{
+    public TransportType TransportType { get; set; }
+    public int Port { get; set; }                  // for TCP/UDP/TLS
+    public bool Enabled { get; set; } = true;
+
+    // TLS-specific
+    public string? ServerCertificatePath { get; set; }
+    public string? ServerCertificatePassword { get; set; }
+    public bool RequireClientCertificate { get; set; }
+
+    // Rabbit-specific
+    public string? RabbitConnectionString { get; set; }
+}
+
+public sealed class ServiceConfig
+{
+    public string Name { get; set; } = string.Empty;
+    public string DefaultVersion { get; set; } = "1.0.0";
+    public IList<string> NeighborRegions { get; set; } = new List<string>();
+}
+```
+
+Use the `PayloadLimits` class from Common (or mirror it here and keep a single definition).
+
+### 1.2 YAML shape
+
+Decide and document a YAML layout, e.g.:
+
+```yaml
+node:
+  nodeId: "gw-eu1-01"
+  region: "eu1"
+  environment: "prod"
+
+payloadLimits:
+  maxRequestBytesPerCall: 10485760      # 10 MB
+  maxRequestBytesPerConnection: 52428800
+  maxAggregateInflightBytes: 209715200
+
+transports:
+  - transportType: Tcp
+    port: 45000
+    enabled: true
+  - transportType: Certificate
+    port: 45001
+    enabled: false
+    serverCertificatePath: "certs/router.pfx"
+    serverCertificatePassword: "secret"
+  - transportType: Udp
+    port: 45002
+    enabled: true
+  - transportType: RabbitMq
+    enabled: true
+    rabbitConnectionString: "amqp://guest:guest@localhost:5672"
+
+services:
+  - name: "Billing"
+    defaultVersion: "1.0.0"
+    neighborRegions: ["eu2", "us1"]
+  - name: "Identity"
+    defaultVersion: "2.1.0"
+    neighborRegions: ["eu2"]
+```
+
+This YAML is the canonical config for the router; environment variables and JSON can override individual properties later via `IConfiguration`.
+
+---
+
+## 2. Implement Router.Config loader and DI extensions
+
+**Project:** `StellaOps.Router.Config`
+
+### 2.1 Choose YAML library
+
+Add a YAML library (e.g. YamlDotNet) to `StellaOps.Router.Config`:
+
+```bash
+dotnet add src/__Libraries/StellaOps.Router.Config/StellaOps.Router.Config.csproj package YamlDotNet
+```
+
+### 2.2 Implement simple loader
+
+Provide a helper that can load YAML into `RouterConfig`:
+
+```csharp
+public static class RouterConfigLoader
+{
+    public static RouterConfig LoadFromYaml(string path)
+    {
+        using var reader = new StreamReader(path);
+        var yaml = new YamlStream();
+        yaml.Load(reader);
+
+        var root = (YamlMappingNode)yaml.Documents[0].RootNode;
+        var json = ConvertYamlToJson(root); // simplest: walk node, serialize to JSON string
+        return JsonSerializer.Deserialize<RouterConfig>(json)!;
+    }
+}
+```
+
+Alternatively, bind YAML directly to `RouterConfig` with YamlDotNet’s object mapping; the detail is implementation‑specific.
+
+### 2.3 ASP.NET Core integration extension
+
+In the router library, add a DI extension the gateway can call:
+
+```csharp
+public static class ServiceCollectionExtensions
+{
+    public static IServiceCollection AddRouterConfig(
+        this IServiceCollection services,
+        IConfiguration configuration)
+    {
+        services.Configure<RouterConfig>(configuration.GetSection("Router"));
+        services.AddSingleton(sp => sp.GetRequiredService<IOptionsMonitor<RouterConfig>>());
+
+        return services;
+    }
+}
+```
+
+Gateway will:
+
+* Add the YAML file to the configuration builder.
+* Call `AddRouterConfig` to bind it.
+
+---
+
+## 3. Wire RouterConfig into Gateway startup & components
+
+**Project:** `StellaOps.Gateway.WebService`
+**Owner:** gateway dev
+
+### 3.1 Program.cs configuration
+
+Adjust `Program.cs`:
+
+```csharp
+var builder = WebApplication.CreateBuilder(args);
+
+// add YAML config
+builder.Configuration
+    .AddJsonFile("appsettings.json", optional: true)
+    .AddYamlFile("router.yaml", optional: false, reloadOnChange: true)
+    .AddEnvironmentVariables("STELLAOPS_");
+
+// bind RouterConfig
+builder.Services.AddRouterConfig(builder.Configuration.GetSection("Router"));
+
+var app = builder.Build();
+```
+
+Key points:
+
+* `AddYamlFile("router.yaml", reloadOnChange: true)` ensures hot‑reload from YAML.
+* `AddEnvironmentVariables("STELLAOPS_")` allows env‑based overrides (optional, but useful).
+
+### 3.2 Inject config into transport factories and routing
+
+Where you start transports:
+
+* Inject `IOptionsMonitor<RouterConfig>` into your `ITransportServerFactory`, and use `RouterConfig.Transports` to know which servers to create and on which ports.
+
+Where you need node identity:
+
+* Inject `IOptionsMonitor<RouterConfig>` into any service needing `GatewayNodeConfig` (e.g. when building `RoutingContext.GatewayRegion`):
+
+  ```csharp
+  var nodeRegion = routerConfig.CurrentValue.Node.Region;
+  ```
+
+Where you need payload limits:
+
+* Inject `IOptionsMonitor<RouterConfig>` into `IPayloadBudget` or `TransportDispatchMiddleware` to fetch current `PayloadLimits`.
+
+Because you’re using `IOptionsMonitor`, components can react to changes when `router.yaml` is modified.
+
+---
+
+## 4. Microservice YAML: schema & loader
+
+**Project:** `__Libraries/StellaOps.Microservice`
+**Owner:** SDK dev
+
+Microservice YAML is optional and used **only** to override endpoint metadata, not to define identity or router pool.
+
+### 4.1 Define YAML shape
+
+Keep it focused on endpoints and overrides:
+
+```yaml
+service:
+  serviceName: "Billing"
+  version: "1.0.0"
+  region: "eu1"
+
+endpoints:
+  - method: "POST"
+    path: "/billing/invoices/upload"
+    defaultTimeout: "00:02:00"
+    supportsStreaming: true
+    requiringClaims:
+      - type: "role"
+        value: "billing-editor"
+  - method: "GET"
+    path: "/billing/invoices/{id}"
+    defaultTimeout: "00:00:10"
+    requiringClaims:
+      - type: "role"
+        value: "billing-reader"
+```
+
+Identity (`serviceName`, `version`, `region`) in YAML is **informative**; the authoritative values still come from `StellaMicroserviceOptions`. If they differ, you log, but don’t override options from YAML.
+
+### 4.2 C# model
+
+In `StellaOps.Microservice`:
+
+```csharp
+internal sealed class MicroserviceYamlConfig
+{
+    public MicroserviceYamlService? Service { get; set; }
+    public IList<MicroserviceYamlEndpoint> Endpoints { get; set; } = new List<MicroserviceYamlEndpoint>();
+}
+
+internal sealed class MicroserviceYamlService
+{
+    public string? ServiceName { get; set; }
+    public string? Version { get; set; }
+    public string? Region { get; set; }
+}
+
+internal sealed class MicroserviceYamlEndpoint
+{
+    public string Method { get; set; } = string.Empty;
+    public string Path { get; set; } = string.Empty;
+    public string? DefaultTimeout { get; set; }
+    public bool? SupportsStreaming { get; set; }
+    public IList<ClaimRequirement> RequiringClaims { get; set; } = new List<ClaimRequirement>();
+}
+```
+
+### 4.3 YAML loader
+
+Reuse YamlDotNet (add package to `StellaOps.Microservice` if needed):
+
+```csharp
+internal interface IMicroserviceYamlLoader
+{
+    MicroserviceYamlConfig? Load(string? path);
+}
+
+internal sealed class MicroserviceYamlLoader : IMicroserviceYamlLoader
+{
+    private readonly ILogger<MicroserviceYamlLoader> _logger;
+
+    public MicroserviceYamlLoader(ILogger<MicroserviceYamlLoader> logger)
+    {
+        _logger = logger;
+    }
+
+    public MicroserviceYamlConfig? Load(string? path)
+    {
+        if (string.IsNullOrWhiteSpace(path) || !File.Exists(path))
+            return null;
+
+        try
+        {
+            using var reader = new StreamReader(path);
+            var deserializer = new DeserializerBuilder().Build();
+            return deserializer.Deserialize<MicroserviceYamlConfig>(reader);
+        }
+        catch (Exception ex)
+        {
+            _logger.LogError(ex, "Failed to load microservice YAML from {Path}", path);
+            return null;
+        }
+    }
+}
+```
+
+Register in DI:
+
+```csharp
+services.AddSingleton<IMicroserviceYamlLoader, MicroserviceYamlLoader>();
+```
+
+---
+
+## 5. Merge YAML overrides with reflection-discovered endpoints
+
+**Project:** `StellaOps.Microservice`
+**Owner:** SDK dev
+
+Extend `EndpointCatalog` to apply YAML overrides.
+
+### 5.1 Extend constructor to accept YAML config
+
+Adjust `EndpointCatalog`:
+
+```csharp
+internal sealed class EndpointCatalog : IEndpointCatalog
+{
+    public IReadOnlyList<EndpointDescriptor> Descriptors { get; }
+
+    private readonly Dictionary<(string Method, string Path), EndpointRegistration> _map;
+
+    public EndpointCatalog(
+        IEndpointDiscovery discovery,
+        IMicroserviceYamlLoader yamlLoader,
+        IOptions<StellaMicroserviceOptions> optionsAccessor)
+    {
+        var options = optionsAccessor.Value;
+
+        var registrations = discovery.DiscoverEndpoints(options);
+        var yamlConfig = yamlLoader.Load(options.ConfigFilePath);
+
+        registrations = ApplyYamlOverrides(registrations, yamlConfig);
+
+        _map = registrations.ToDictionary(
+            r => (r.Descriptor.Method, r.Descriptor.Path),
+            r => r,
+            StringComparer.OrdinalIgnoreCase);
+
+        Descriptors = registrations.Select(r => r.Descriptor).ToArray();
+    }
+}
+```
+
+### 5.2 Implement `ApplyYamlOverrides`
+
+Key rules:
+
+* Identity (ServiceName, Version, Region) always come from `StellaMicroserviceOptions`.
+* YAML can override:
+
+  * `DefaultTimeout`
+  * `SupportsStreaming`
+  * `RequiringClaims`
+
+Implementation sketch:
+
+```csharp
+private static IReadOnlyList<EndpointRegistration> ApplyYamlOverrides(
+    IReadOnlyList<EndpointRegistration> registrations,
+    MicroserviceYamlConfig? yaml)
+{
+    if (yaml is null || yaml.Endpoints.Count == 0)
+        return registrations;
+
+    var overrideMap = yaml.Endpoints.ToDictionary(
+        e => (e.Method, e.Path),
+        e => e,
+        StringComparer.OrdinalIgnoreCase);
+
+    var result = new List<EndpointRegistration>(registrations.Count);
+
+    foreach (var reg in registrations)
+    {
+        if (!overrideMap.TryGetValue((reg.Descriptor.Method, reg.Descriptor.Path), out var ov))
+        {
+            result.Add(reg);
+            continue;
+        }
+
+        var desc = reg.Descriptor;
+
+        var timeout = desc.DefaultTimeout;
+        if (!string.IsNullOrWhiteSpace(ov.DefaultTimeout) &&
+            TimeSpan.TryParse(ov.DefaultTimeout, out var parsed))
+        {
+            timeout = parsed;
+        }
+
+        var supportsStreaming = desc.SupportsStreaming;
+        if (ov.SupportsStreaming.HasValue)
+        {
+            supportsStreaming = ov.SupportsStreaming.Value;
+        }
+
+        var requiringClaims = ov.RequiringClaims.Count > 0
+            ? ov.RequiringClaims.ToArray()
+            : desc.RequiringClaims;
+
+        var overriddenDescriptor = new EndpointDescriptor
+        {
+            ServiceName = desc.ServiceName,
+            Version = desc.Version,
+            Method = desc.Method,
+            Path = desc.Path,
+            DefaultTimeout = timeout,
+            SupportsStreaming = supportsStreaming,
+            RequiringClaims = requiringClaims
+        };
+
+        result.Add(new EndpointRegistration
+        {
+            Descriptor = overriddenDescriptor,
+            HandlerType = reg.HandlerType
+        });
+    }
+
+    return result;
+}
+```
+
+This ensures code defines the set of endpoints; YAML only tunes metadata.
+
+---
+
+## 6. Hot‑reload / YAML change handling
+
+**Router side:** you already enabled `reloadOnChange` for `router.yaml`, and use `IOptionsMonitor<RouterConfig>`. Next:
+
+* Components that care about changes must **react**:
+
+  * Payload limits:
+
+    * `IPayloadBudget` or `TransportDispatchMiddleware` should read `routerConfig.CurrentValue.PayloadLimits` on each request rather than caching.
+  * Node region:
+
+    * `RoutingContext.GatewayRegion` can be built from `routerConfig.CurrentValue.Node.Region` per request.
+
+You do **not** need a custom watcher; `IOptionsMonitor` already tracks config changes.
+
+**Microservice side:** for now you can start with **load-on-startup** YAML. If you want hot‑reload:
+
+* Implement a FileSystemWatcher in `MicroserviceYamlLoader` or a small `IHostedService`:
+
+  * Watch `options.ConfigFilePath` for changes.
+  * On change:
+
+    * Reload YAML.
+    * Rebuild `EndpointDescriptor` list.
+    * Send an updated HELLO or an ENDPOINTS_UPDATE frame to router.
+
+Given complexity, you can postpone true hot reload to a later iteration and document that microservices must be restarted to pick up YAML changes.
+
+---
+
+## 7. Tests
+
+**Router.Config tests:**
+
+* Unit tests for `RouterConfigLoader`:
+
+  * Given a YAML string, bind to `RouterConfig` properly.
+  * Validate `TransportType.Tcp` / `Udp` / `RabbitMq` values map correctly.
+
+* Integration test:
+
+  * Start gateway with `router.yaml`.
+  * Access `IOptionsMonitor<RouterConfig>` in a test controller or test service and assert values.
+  * Modify YAML on disk (if test infra allows) and ensure values update via `IOptionsMonitor`.
+
+**Microservice YAML tests:**
+
+* Unit tests for `MicroserviceYamlLoader`:
+
+  * Load valid YAML, confirm endpoints and claims/timeouts parsed.
+
+* `EndpointCatalog` tests:
+
+  * Build fake `EndpointRegistration` list from reflection.
+  * Build YAML overrides.
+  * Call `ApplyYamlOverrides` and assert:
+
+    * Timeouts updated.
+    * SupportsStreaming updated.
+    * RequiringClaims replaced where provided.
+    * Descriptors with no matching YAML remain unchanged.
+
+---
+
+## 8. Documentation updates
+
+Update docs under `docs/router`:
+
+1. **Stella Ops Router – Webserver.md**:
+
+   * Describe `router.yaml`:
+
+     * Node config (region, nodeId).
+     * PayloadLimits.
+     * Transports.
+   * Explain precedence:
+
+     * YAML as base.
+     * Environment variables can override individual fields via `STELLAOPS_Router__Node__Region` etc.
+
+2. **Stella Ops Router – Microservice.md**:
+
+   * Explain `ConfigFilePath` in `StellaMicroserviceOptions`.
+   * Show full example microservice YAML and how it maps to endpoint metadata.
+   * Clearly state:
+
+     * Identity comes from options (code/config), not YAML.
+     * YAML can override per‑endpoint timeout, streaming flag, requiringClaims.
+     * YAML can’t add endpoints that don’t exist in code.
+
+3. **Stella Ops Router Documentation.md**:
+
+   * Add a short “Configuration” chapter:
+
+     * Where `router.yaml` lives.
+     * Where microservice YAML lives.
+     * How to run locally with custom configs.
+
+---
+
+## 9. Done criteria for “Add Router.Config + Microservice YAML integration”
+
+You can call step 10 complete when:
+
+* Router:
+
+  * Loads `router.yaml` into `RouterConfig` using `StellaOps.Router.Config`.
+  * Uses `RouterConfig.Node.Region` when building routing context.
+  * Uses `RouterConfig.PayloadLimits` for payload budget enforcement.
+  * Uses `RouterConfig.Transports` to start the right `ITransportServer` instances.
+  * Supports runtime changes to `router.yaml` via `IOptionsMonitor` for at least node identity and payload limits.
+
+* Microservice:
+
+  * Accepts optional `ConfigFilePath` in `StellaMicroserviceOptions`.
+  * Loads YAML (when present) and merges overrides into reflection‑discovered endpoints.
+  * Sends HELLO with the **merged** descriptors (i.e., YAML-aware defaults).
+  * Behavior remains unchanged when no YAML is provided (pure reflection mode).
+
+* Tests:
+
+  * Confirm config binding for router and microservice.
+  * Confirm YAML overrides are applied correctly to endpoint metadata.
+
+At that point, configuration is no longer hardcoded, and you have a clear, documented path for both router operators and microservice teams to configure behavior via YAML with predictable precedence.