563 lines
16 KiB
Markdown
563 lines
16 KiB
Markdown
For this step you’re taking the protocol you already proved with InMemory and putting it on real transports:
|
||
|
||
* TCP (baseline)
|
||
* Certificate/TLS (secure TCP)
|
||
* UDP (small, non‑streaming)
|
||
* RabbitMQ
|
||
|
||
The idea: every plugin implements the same `Frame` semantics (HELLO/HEARTBEAT/REQUEST/RESPONSE/CANCEL, plus streaming where supported), and the gateway/microservices don’t change their business logic at all.
|
||
|
||
I’ll structure this as a sequence of sub‑steps you can execute in order.
|
||
|
||
---
|
||
|
||
## 0. Preconditions
|
||
|
||
Before you start adding real transports, make sure:
|
||
|
||
* Frame model is stable in `StellaOps.Router.Common`:
|
||
|
||
* `Frame`, `FrameType`, `TransportType`.
|
||
* Microservice and gateway code use **only**:
|
||
|
||
* `ITransportClient` to send (gateway side).
|
||
* `ITransportServer` / connection abstractions to receive (gateway side).
|
||
* `IMicroserviceConnection` + `ITransportClient` under the hood (microservice side).
|
||
* InMemory transport is working with:
|
||
|
||
* HELLO
|
||
* REQUEST / RESPONSE
|
||
* CANCEL
|
||
* Streaming & payload limits (step 8)
|
||
|
||
If any code still directly talks to “InMemoryRouterHub” from app logic, hide it behind the `ITransportClient` / `ITransportServer` abstractions first.
|
||
|
||
---
|
||
|
||
## 1. Freeze the wire protocol and serializer
|
||
|
||
**Owner:** protocol / infra dev
|
||
|
||
Before touching sockets or RabbitMQ, lock down **how a `Frame` is encoded** on the wire. This must be consistent across all transports except InMemory (which can cheat a bit internally).
|
||
|
||
### 1.1 Frame header
|
||
|
||
Define a simple binary header; for example:
|
||
|
||
* 1 byte: `FrameType`
|
||
* 16 bytes: `CorrelationId` (`Guid`)
|
||
* 4 bytes: payload length (`int32`, big- or little-endian, but be consistent)
|
||
|
||
Total header = 21 bytes. Then `payloadLength` bytes follow.
|
||
|
||
You can evolve later but start with something simple.
|
||
|
||
### 1.2 Frame serializer
|
||
|
||
In a small shared, **non‑ASP.NET** assembly (either Common or a new `StellaOps.Router.Protocol` library), implement:
|
||
|
||
```csharp
|
||
public interface IFrameSerializer
|
||
{
|
||
void WriteFrame(Frame frame, Stream stream, CancellationToken ct);
|
||
Task WriteFrameAsync(Frame frame, Stream stream, CancellationToken ct);
|
||
|
||
Frame ReadFrame(Stream stream, CancellationToken ct);
|
||
Task<Frame> ReadFrameAsync(Stream stream, CancellationToken ct);
|
||
}
|
||
```
|
||
|
||
Implementation:
|
||
|
||
* Writes header then payload.
|
||
* Reads header then payload; throws on EOF.
|
||
|
||
For payloads (HELLO, HEARTBEAT, etc.), use one encoding consistently (e.g. `System.Text.Json` for now) and **centralize** DTO ⇒ `byte[]` conversions:
|
||
|
||
```csharp
|
||
public static class PayloadCodec
|
||
{
|
||
public static byte[] Encode<T>(T payload) { ... }
|
||
public static T Decode<T>(byte[] bytes) { ... }
|
||
}
|
||
```
|
||
|
||
All transports use `IFrameSerializer` + `PayloadCodec`.
|
||
|
||
---
|
||
|
||
## 2. Introduce a transport registry / resolver
|
||
|
||
**Projects:** gateway + microservice
|
||
**Owner:** infra dev
|
||
|
||
You need a way to map `TransportType` to a concrete plugin.
|
||
|
||
### 2.1 Gateway side
|
||
|
||
Define:
|
||
|
||
```csharp
|
||
public interface ITransportClientResolver
|
||
{
|
||
ITransportClient GetClient(TransportType transportType);
|
||
}
|
||
|
||
public interface ITransportServerFactory
|
||
{
|
||
ITransportServer CreateServer(TransportType transportType);
|
||
}
|
||
```
|
||
|
||
Initial implementation:
|
||
|
||
* Registers the available clients:
|
||
|
||
```csharp
|
||
public sealed class TransportClientResolver : ITransportClientResolver
|
||
{
|
||
private readonly IServiceProvider _sp;
|
||
|
||
public TransportClientResolver(IServiceProvider sp) => _sp = sp;
|
||
|
||
public ITransportClient GetClient(TransportType transportType) =>
|
||
transportType switch
|
||
{
|
||
TransportType.Tcp => _sp.GetRequiredService<TcpTransportClient>(),
|
||
TransportType.Certificate=> _sp.GetRequiredService<TlsTransportClient>(),
|
||
TransportType.Udp => _sp.GetRequiredService<UdpTransportClient>(),
|
||
TransportType.RabbitMq => _sp.GetRequiredService<RabbitMqTransportClient>(),
|
||
_ => throw new NotSupportedException($"Transport {transportType} not supported.")
|
||
};
|
||
}
|
||
```
|
||
|
||
Then in `TransportDispatchMiddleware`, instead of injecting a single `ITransportClient`, inject `ITransportClientResolver` and choose:
|
||
|
||
```csharp
|
||
var client = clientResolver.GetClient(decision.TransportType);
|
||
```
|
||
|
||
### 2.2 Microservice side
|
||
|
||
On the microservice, you can do something similar:
|
||
|
||
```csharp
|
||
internal interface IMicroserviceTransportConnector
|
||
{
|
||
Task ConnectAsync(StellaMicroserviceOptions options, CancellationToken ct);
|
||
}
|
||
```
|
||
|
||
Implement one per transport type; later `StellaMicroserviceOptions.Routers` will determine which transport to use for each router endpoint.
|
||
|
||
---
|
||
|
||
## 3. Implement plugin 1: TCP
|
||
|
||
Start with TCP; it’s the most straightforward and will largely mirror your InMemory behavior.
|
||
|
||
### 3.1 Gateway: `TcpTransportServer`
|
||
|
||
**Project:** `StellaOps.Gateway.WebService` or a transport sub-namespace.
|
||
|
||
Responsibilities:
|
||
|
||
* Listen on a configured TCP port (e.g. from `RouterConfig`).
|
||
* Accept connections, each mapping to a `ConnectionId`.
|
||
* For each connection:
|
||
|
||
* Start a background receive loop:
|
||
|
||
* Use `IFrameSerializer.ReadFrameAsync` on a `NetworkStream`.
|
||
* On `FrameType.Hello`:
|
||
|
||
* Deserialize HELLO payload.
|
||
* Build a `ConnectionState` and register with `IGlobalRoutingState`.
|
||
* On `FrameType.Heartbeat`:
|
||
|
||
* Update heartbeat for that `ConnectionId`.
|
||
* On `FrameType.Response` or `ResponseStreamData`:
|
||
|
||
* Push frame into the gateway’s correlation / streaming handler (similar to InMemory path).
|
||
* On `FrameType.Cancel` (rare from microservice):
|
||
|
||
* Optionally implement; can be ignored for now.
|
||
* Provide a sending API to the matching `TcpTransportClient` (gateway-side) using `WriteFrameAsync`.
|
||
|
||
You will likely have:
|
||
|
||
* A `TcpConnectionContext` per connected microservice:
|
||
|
||
* Holds `ConnectionId`, `TcpClient`, `NetworkStream`, `TaskCompletionSource` maps for correlation IDs.
|
||
|
||
### 3.2 Gateway: `TcpTransportClient` (gateway-side, to microservices)
|
||
|
||
Implements `ITransportClient`:
|
||
|
||
* `SendRequestAsync`:
|
||
|
||
* Given `ConnectionState`:
|
||
|
||
* Get the associated `TcpConnectionContext`.
|
||
* Register a `TaskCompletionSource<Frame>` keyed by `CorrelationId`.
|
||
* Call `WriteFrameAsync(requestFrame)` on the connection’s stream.
|
||
* Await the TCS, which is completed in the receive loop when a `Response` frame arrives.
|
||
* `SendStreamingAsync`:
|
||
|
||
* Write header `FrameType.Request`.
|
||
* Read from `BudgetedRequestStream` in chunks:
|
||
|
||
* For TCP plugin you can either:
|
||
|
||
* Use `RequestStreamData` frames with chunk payloads, or
|
||
* Keep the simple bridging approach and send a single `Request` with all body bytes.
|
||
* Since you already validated streaming semantics with InMemory, you can decide:
|
||
|
||
* For first version of TCP, **only support buffered data**, then add chunk frames later.
|
||
* `SendCancelAsync`:
|
||
|
||
* Write a `FrameType.Cancel` frame with the same `CorrelationId`.
|
||
|
||
### 3.3 Microservice: `TcpTransportClientConnection`
|
||
|
||
**Project:** `StellaOps.Microservice`
|
||
|
||
Responsibilities on microservice side:
|
||
|
||
* For each `RouterEndpointConfig` where `TransportType == Tcp`:
|
||
|
||
* Open a `TcpClient` to `Host:Port`.
|
||
* Use `IFrameSerializer` to send:
|
||
|
||
* `HELLO` frame (payload = identity + descriptors).
|
||
* Periodic `HEARTBEAT` frames.
|
||
* `RESPONSE` frames for incoming `REQUEST`s.
|
||
|
||
* Receive loop:
|
||
|
||
* `ReadFrameAsync` from `NetworkStream`.
|
||
* On `REQUEST`:
|
||
|
||
* Dispatch through `IEndpointDispatcher`.
|
||
* For minimal streaming, treat payload as buffered; you’ll align with streaming later.
|
||
* On `CANCEL`:
|
||
|
||
* Use correlation ID to cancel the `CancellationTokenSource` you already maintain.
|
||
|
||
This is conceptually the same as InMemory but using real sockets.
|
||
|
||
---
|
||
|
||
## 4. Implement plugin 2: Certificate/TLS
|
||
|
||
Build TLS on top of TCP plugin; do not fork logic unnecessarily.
|
||
|
||
### 4.1 Gateway: `TlsTransportServer`
|
||
|
||
* Wrap accepted `TcpClient` sockets in `SslStream`.
|
||
* Load server certificate from configuration (for the node/region).
|
||
* Authenticate client if you want mutual TLS.
|
||
|
||
Structure:
|
||
|
||
* Reuse almost all of `TcpTransportServer` logic, but instead of `NetworkStream` you use `SslStream` as the underlying stream for `IFrameSerializer`.
|
||
|
||
### 4.2 Microservice: `TlsTransportClientConnection`
|
||
|
||
* Instead of plain `TcpClient.GetStream`, wrap in `SslStream`.
|
||
* Authenticate server (hostname & certificate).
|
||
* Optional: present client certificate.
|
||
|
||
Configuration fields in `RouterEndpointConfig` (or a TLS-specific sub-config):
|
||
|
||
* `UseTls` / `TransportType.Certificate`.
|
||
* Certificate paths / thumbprints / validation parameters.
|
||
|
||
At the SDK level, you just treat it as a different transport type; protocol remains identical.
|
||
|
||
---
|
||
|
||
## 5. Implement plugin 3: UDP (small, non‑streaming)
|
||
|
||
UDP is only for small, bounded payloads. No streaming, best‑effort delivery.
|
||
|
||
### 5.1 Constraints
|
||
|
||
* Use UDP **only** for buffered, small payload endpoints.
|
||
* No streaming (`SupportsStreaming` must be `false` for UDP endpoints).
|
||
* No guarantee of delivery or ordering; caller must tolerate occasional failures/timeouts.
|
||
|
||
### 5.2 Gateway: `UdpTransportServer`
|
||
|
||
Responsibilities:
|
||
|
||
* Listen on a UDP port.
|
||
* Parse each incoming datagram as a full `Frame`:
|
||
|
||
* `FrameType.Hello`:
|
||
|
||
* Register a “logical connection” keyed by `(remoteEndpoint)` and `InstanceId`.
|
||
* `FrameType.Heartbeat`:
|
||
|
||
* Update health for that logical connection.
|
||
* `FrameType.Response`:
|
||
|
||
* Use `CorrelationId` and “connectionId” to complete a `TaskCompletionSource` as with TCP.
|
||
|
||
Because UDP is connectionless, your `ConnectionId` can be:
|
||
|
||
* A composite of microservice identity + remote endpoint, e.g. `"{instanceId}@{ip}:{port}"`.
|
||
|
||
### 5.3 Gateway: `UdpTransportClient` (gateway-side)
|
||
|
||
`SendRequestAsync`:
|
||
|
||
* Serialize `Frame` to `byte[]`.
|
||
* Send via `UdpClient.SendAsync` to the remote endpoint from `ConnectionState`.
|
||
* Start a timer:
|
||
|
||
* Wait for `Response` datagram with matching `CorrelationId`.
|
||
* If none comes within timeout → throw `OperationCanceledException`.
|
||
|
||
`SendStreamingAsync`:
|
||
|
||
* For this first iteration, **throw NotSupportedException**.
|
||
* Router should not route streaming endpoints over UDP; your routing config should enforce that.
|
||
|
||
`SendCancelAsync`:
|
||
|
||
* Optionally send a CANCEL datagram; but in practice, if requests are small, this is less useful. You can still implement it for symmetry.
|
||
|
||
### 5.4 Microservice: UDP connection
|
||
|
||
For microservice side:
|
||
|
||
* A single `UdpClient` bound to a local port.
|
||
* For each configured router (host/port):
|
||
|
||
* HELLO: send a `FrameType.Hello` datagram.
|
||
* HEARTBEAT: send periodic `FrameType.Heartbeat`.
|
||
* REQUEST handling: not needed; UDP plugin is used **for gateway → microservice** only if you design it that way. More likely, microservice is the server in TCP, but for UDP you might decide microservice is listening on port and gateway sends requests. So invert roles if needed.
|
||
|
||
Given the complexity and limited utility, you can treat UDP as “advanced/optional transport” and implement it last.
|
||
|
||
---
|
||
|
||
## 6. Implement plugin 4: RabbitMQ
|
||
|
||
This is conceptually similar to what you had in Serdica.
|
||
|
||
### 6.1 Exchange/queue design
|
||
|
||
Decide and document (in `Protocol & Transport Specification.md`) something like:
|
||
|
||
* Exchange: `stella.router`
|
||
* Routing keys:
|
||
|
||
* `request.{serviceName}.{version}` — gateway → microservice.
|
||
* Microservice’s reply queue per instance: `reply.{serviceName}.{version}.{instanceId}`.
|
||
|
||
Rabbit usages:
|
||
|
||
* Gateway:
|
||
|
||
* Publishes REQUEST frames to `request.{serviceName}.{version}`.
|
||
* Consumes from `reply.*` for responses.
|
||
|
||
* Microservice:
|
||
|
||
* Consumes from `request.{serviceName}.{version}`.
|
||
* Publishes responses to its own reply queue; sets `CorrelationId` property.
|
||
|
||
### 6.2 Gateway: `RabbitMqTransportClient`
|
||
|
||
Implements `ITransportClient`:
|
||
|
||
* `SendRequestAsync`:
|
||
|
||
* Create a message with:
|
||
|
||
* Body = serialized `Frame` (REQUEST or buffered streaming).
|
||
* Properties:
|
||
|
||
* `CorrelationId` = `frame.CorrelationId`.
|
||
* `ReplyTo` = microservice’s reply queue name for this instance.
|
||
* Publish to `request.{serviceName}.{version}`.
|
||
* Await a response:
|
||
|
||
* Consumer on reply queue completes a `TaskCompletionSource<Frame>` keyed by correlation ID.
|
||
|
||
* `SendStreamingAsync`:
|
||
|
||
* For v1, you can:
|
||
|
||
* Only support buffered endpoints over RabbitMQ (like UDP).
|
||
* Or send chunked messages (`RequestStreamData` frames as separate messages) and reconstruct on microservice side.
|
||
* I’d recommend:
|
||
|
||
* Start with buffered only over RabbitMQ.
|
||
* Mark Rabbit as “no streaming support yet” in config.
|
||
|
||
* `SendCancelAsync`:
|
||
|
||
* Option 1: send a separate CANCEL message with same `CorrelationId`.
|
||
* Option 2: rely on timeout; cancellation doesn’t buy much given overhead.
|
||
|
||
### 6.3 Microservice: RabbitMQ listener
|
||
|
||
* Single `IConnection` and `IModel`.
|
||
|
||
* Declare and bind:
|
||
|
||
* Service request queue: `request.{serviceName}.{version}`.
|
||
* Reply queue: `reply.{serviceName}.{version}.{instanceId}`.
|
||
|
||
* Consume request queue:
|
||
|
||
* On message:
|
||
|
||
* Deserialize `Frame`.
|
||
* Dispatch through `IEndpointDispatcher`.
|
||
* Publish RESPONSE message to `ReplyTo` queue with same `CorrelationId`.
|
||
|
||
If you already have RabbitMQ experience from Serdica, this should feel familiar.
|
||
|
||
---
|
||
|
||
## 7. Routing config & transport selection
|
||
|
||
**Projects:** router config + microservice options
|
||
**Owner:** config / platform dev
|
||
|
||
You need to define which transport is actually used in production.
|
||
|
||
### 7.1 Gateway config (RouterConfig)
|
||
|
||
Per service/instance, store:
|
||
|
||
* `TransportType` to listen on / expect connections for.
|
||
* Ports / Rabbit URLs / TLS settings.
|
||
|
||
Example shape in `RouterConfig`:
|
||
|
||
```csharp
|
||
public sealed class ServiceInstanceConfig
|
||
{
|
||
public string ServiceName { get; set; } = string.Empty;
|
||
public string Version { get; set; } = string.Empty;
|
||
public string Region { get; set; } = string.Empty;
|
||
public TransportType TransportType { get; set; } = TransportType.Udp; // default
|
||
public int Port { get; set; } // for TCP/UDP/TLS
|
||
public string? RabbitConnectionString { get; set; }
|
||
// TLS info, etc.
|
||
}
|
||
```
|
||
|
||
`StellaOps.Gateway.WebService` startup:
|
||
|
||
* Reads these configs.
|
||
* Starts corresponding `ITransportServer` instances.
|
||
|
||
### 7.2 Microservice options
|
||
|
||
`StellaMicroserviceOptions.Routers` entries must define:
|
||
|
||
* `Host`
|
||
* `Port`
|
||
* `TransportType`
|
||
* Any transport-specific settings (TLS, Rabbit URL).
|
||
|
||
At connect time, microservice chooses:
|
||
|
||
* For each `RouterEndpointConfig`, instantiate the right connector:
|
||
|
||
```csharp
|
||
switch(config.TransportType)
|
||
{
|
||
case TransportType.Tcp:
|
||
use TcpMicroserviceConnector;
|
||
break;
|
||
case TransportType.Certificate:
|
||
use TlsMicroserviceConnector;
|
||
break;
|
||
case TransportType.Udp:
|
||
use UdpMicroserviceConnector;
|
||
break;
|
||
case TransportType.RabbitMq:
|
||
use RabbitMqMicroserviceConnector;
|
||
break;
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 8. Implementation order & testing strategy
|
||
|
||
**Owner:** tech lead
|
||
|
||
Do NOT try to implement all at once. Suggested order:
|
||
|
||
1. **TCP**:
|
||
|
||
* Reuse InMemory test suite:
|
||
|
||
* HELLO + endpoint registration.
|
||
* REQUEST → RESPONSE.
|
||
* CANCEL.
|
||
* Heartbeats.
|
||
* (Optional) streaming as buffered stub for v1, then add genuine streaming.
|
||
|
||
2. **Certificate/TLS**:
|
||
|
||
* Wrap TCP logic in TLS.
|
||
* Same tests, plus:
|
||
|
||
* Certificate validation.
|
||
* Mutual TLS if required.
|
||
|
||
3. **RabbitMQ**:
|
||
|
||
* Start with buffered-only endpoints.
|
||
* Mirror existing InMemory/TCP tests where payloads are small.
|
||
* Add tests for connection resilience (reconnect, etc.).
|
||
|
||
4. **UDP**:
|
||
|
||
* Implement only for very small buffered requests; no streaming.
|
||
* Add tests that verify:
|
||
|
||
* HELLO + basic health.
|
||
* REQUEST → RESPONSE with small payload.
|
||
* Proper timeouts.
|
||
|
||
At each stage, tests for that plugin must reuse the **same microservice and gateway** code that worked with InMemory. Only the transport factories change.
|
||
|
||
---
|
||
|
||
## 9. Done criteria for “Implement real transport plugins one by one”
|
||
|
||
You can consider step 9 done when:
|
||
|
||
* There are **concrete implementations** of `ITransportServer` + `ITransportClient` for:
|
||
|
||
* TCP
|
||
* Certificate/TLS
|
||
* UDP (buffered only)
|
||
* RabbitMQ (buffered at minimum)
|
||
* Gateway startup:
|
||
|
||
* Reads `RouterConfig`.
|
||
* Starts appropriate transport servers per node/region.
|
||
* Microservice SDK:
|
||
|
||
* Reads `StellaMicroserviceOptions.Routers`.
|
||
* Connects to router nodes using the configured `TransportType`.
|
||
* Uses the same HELLO/HEARTBEAT/REQUEST/RESPONSE/CANCEL semantics as InMemory.
|
||
* The same functional tests that passed for InMemory:
|
||
|
||
* Now pass with TCP plugin.
|
||
* At least a subset pass with TLS, Rabbit, and UDP, honoring their constraints (no streaming on UDP, etc.).
|
||
|
||
From there, you can move into hardening each plugin (reconnect, backoff, error handling) and documenting “which transport to use when” in your router docs.
|