git.stella-ops.org/docs/router/09-Step.md

For this step you’re taking the protocol you already proved with InMemory and putting it on real transports:

• TCP (baseline)
• Certificate/TLS (secure TCP)
• UDP (small, non‑streaming)
• RabbitMQ

The idea: every plugin implements the same Frame semantics (HELLO/HEARTBEAT/REQUEST/RESPONSE/CANCEL, plus streaming where supported), and the gateway/microservices don’t change their business logic at all.

I’ll structure this as a sequence of sub‑steps you can execute in order.

---

0. Preconditions

Before you start adding real transports, make sure:

• Frame model is stable in StellaOps.Router.Common:

  • Frame, FrameType, TransportType.

• Microservice and gateway code use only:

  • ITransportClient to send (gateway side).
  • ITransportServer / connection abstractions to receive (gateway side).
  • IMicroserviceConnection + ITransportClient under the hood (microservice side).

• InMemory transport is working with:

  • HELLO
  • REQUEST / RESPONSE
  • CANCEL
  • Streaming & payload limits (step 8)

If any code still directly talks to “InMemoryRouterHub” from app logic, hide it behind the ITransportClient / ITransportServer abstractions first.

---

1. Freeze the wire protocol and serializer

Owner: protocol / infra dev

Before touching sockets or RabbitMQ, lock down how a Frame is encoded on the wire. This must be consistent across all transports except InMemory (which can cheat a bit internally).

1.1 Frame header

Define a simple binary header; for example:

• 1 byte: FrameType
• 16 bytes: CorrelationId (Guid)
• 4 bytes: payload length (int32, big- or little-endian, but be consistent)

Total header = 21 bytes. Then payloadLength bytes follow.

You can evolve later but start with something simple.

1.2 Frame serializer

In a small shared, non‑ASP.NET assembly (either Common or a new StellaOps.Router.Protocol library), implement:

public interface IFrameSerializer
{
    void WriteFrame(Frame frame, Stream stream, CancellationToken ct);
    Task WriteFrameAsync(Frame frame, Stream stream, CancellationToken ct);

    Frame ReadFrame(Stream stream, CancellationToken ct);
    Task<Frame> ReadFrameAsync(Stream stream, CancellationToken ct);
}

Implementation:

• Writes header then payload.
• Reads header then payload; throws on EOF.

For payloads (HELLO, HEARTBEAT, etc.), use one encoding consistently (e.g. System.Text.Json for now) and centralize DTO ⇒ byte[] conversions:

public static class PayloadCodec
{
    public static byte[] Encode<T>(T payload) { ... }
    public static T Decode<T>(byte[] bytes) { ... }
}

All transports use IFrameSerializer + PayloadCodec.

---

2. Introduce a transport registry / resolver

Projects: gateway + microservice Owner: infra dev

You need a way to map TransportType to a concrete plugin.

2.1 Gateway side

Define:

public interface ITransportClientResolver
{
    ITransportClient GetClient(TransportType transportType);
}

public interface ITransportServerFactory
{
    ITransportServer CreateServer(TransportType transportType);
}

Initial implementation:

• Registers the available clients:

public sealed class TransportClientResolver : ITransportClientResolver
{
    private readonly IServiceProvider _sp;

    public TransportClientResolver(IServiceProvider sp) => _sp = sp;

    public ITransportClient GetClient(TransportType transportType) =>
        transportType switch
        {
            TransportType.Tcp        => _sp.GetRequiredService<TcpTransportClient>(),
            TransportType.Certificate=> _sp.GetRequiredService<TlsTransportClient>(),
            TransportType.Udp        => _sp.GetRequiredService<UdpTransportClient>(),
            TransportType.RabbitMq   => _sp.GetRequiredService<RabbitMqTransportClient>(),
            _ => throw new NotSupportedException($"Transport {transportType} not supported.")
        };
}

Then in TransportDispatchMiddleware, instead of injecting a single ITransportClient, inject ITransportClientResolver and choose:

var client = clientResolver.GetClient(decision.TransportType);

2.2 Microservice side

On the microservice, you can do something similar:

internal interface IMicroserviceTransportConnector
{
    Task ConnectAsync(StellaMicroserviceOptions options, CancellationToken ct);
}

Implement one per transport type; later StellaMicroserviceOptions.Routers will determine which transport to use for each router endpoint.

---

3. Implement plugin 1: TCP

Start with TCP; it’s the most straightforward and will largely mirror your InMemory behavior.

3.1 Gateway: TcpTransportServer

Project: StellaOps.Gateway.WebService or a transport sub-namespace.

Responsibilities:

• Listen on a configured TCP port (e.g. from RouterConfig).

• Accept connections, each mapping to a ConnectionId.

• For each connection:

  • Start a background receive loop:

    • Use IFrameSerializer.ReadFrameAsync on a NetworkStream.

    • On FrameType.Hello:

      • Deserialize HELLO payload.
      • Build a ConnectionState and register with IGlobalRoutingState.

    • On FrameType.Heartbeat:

      • Update heartbeat for that ConnectionId.

    • On FrameType.Response or ResponseStreamData:

      • Push frame into the gateway’s correlation / streaming handler (similar to InMemory path).

    • On FrameType.Cancel (rare from microservice):

      • Optionally implement; can be ignored for now.

  • Provide a sending API to the matching TcpTransportClient (gateway-side) using WriteFrameAsync.

You will likely have:

• A TcpConnectionContext per connected microservice:

  • Holds ConnectionId, TcpClient, NetworkStream, TaskCompletionSource maps for correlation IDs.

3.2 Gateway: TcpTransportClient (gateway-side, to microservices)

Implements ITransportClient:

• SendRequestAsync:

  • Given ConnectionState:

    • Get the associated TcpConnectionContext.
    • Register a TaskCompletionSource<Frame> keyed by CorrelationId.
    • Call WriteFrameAsync(requestFrame) on the connection’s stream.
    • Await the TCS, which is completed in the receive loop when a Response frame arrives.

• SendStreamingAsync:

  • Write header FrameType.Request.

  • Read from BudgetedRequestStream in chunks:

    • For TCP plugin you can either:

      • Use RequestStreamData frames with chunk payloads, or
      • Keep the simple bridging approach and send a single Request with all body bytes.

    • Since you already validated streaming semantics with InMemory, you can decide:

      • For first version of TCP, only support buffered data, then add chunk frames later.

• SendCancelAsync:

  • Write a FrameType.Cancel frame with the same CorrelationId.

3.3 Microservice: TcpTransportClientConnection

Project: StellaOps.Microservice

Responsibilities on microservice side:

• For each RouterEndpointConfig where TransportType == Tcp:

  • Open a TcpClient to Host:Port.

  • Use IFrameSerializer to send:

    • HELLO frame (payload = identity + descriptors).
    • Periodic HEARTBEAT frames.
    • RESPONSE frames for incoming REQUESTs.

• Receive loop:

  • ReadFrameAsync from NetworkStream.

  • On REQUEST:

    • Dispatch through IEndpointDispatcher.
    • For minimal streaming, treat payload as buffered; you’ll align with streaming later.

  • On CANCEL:

    • Use correlation ID to cancel the CancellationTokenSource you already maintain.

This is conceptually the same as InMemory but using real sockets.

---

4. Implement plugin 2: Certificate/TLS

Build TLS on top of TCP plugin; do not fork logic unnecessarily.

4.1 Gateway: TlsTransportServer

• Wrap accepted TcpClient sockets in SslStream.
• Load server certificate from configuration (for the node/region).
• Authenticate client if you want mutual TLS.

Structure:

• Reuse almost all of TcpTransportServer logic, but instead of NetworkStream you use SslStream as the underlying stream for IFrameSerializer.

4.2 Microservice: TlsTransportClientConnection

• Instead of plain TcpClient.GetStream, wrap in SslStream.
• Authenticate server (hostname & certificate).
• Optional: present client certificate.

Configuration fields in RouterEndpointConfig (or a TLS-specific sub-config):

• UseTls / TransportType.Certificate.
• Certificate paths / thumbprints / validation parameters.

At the SDK level, you just treat it as a different transport type; protocol remains identical.

---

5. Implement plugin 3: UDP (small, non‑streaming)

UDP is only for small, bounded payloads. No streaming, best‑effort delivery.

5.1 Constraints

• Use UDP only for buffered, small payload endpoints.
• No streaming (SupportsStreaming must be false for UDP endpoints).
• No guarantee of delivery or ordering; caller must tolerate occasional failures/timeouts.

5.2 Gateway: UdpTransportServer

Responsibilities:

• Listen on a UDP port.

• Parse each incoming datagram as a full Frame:

  • FrameType.Hello:

    • Register a “logical connection” keyed by (remoteEndpoint) and InstanceId.

  • FrameType.Heartbeat:

    • Update health for that logical connection.

  • FrameType.Response:

    • Use CorrelationId and “connectionId” to complete a TaskCompletionSource as with TCP.

Because UDP is connectionless, your ConnectionId can be:

• A composite of microservice identity + remote endpoint, e.g. "{instanceId}@{ip}:{port}".

5.3 Gateway: UdpTransportClient (gateway-side)

SendRequestAsync:

• Serialize Frame to byte[].

• Send via UdpClient.SendAsync to the remote endpoint from ConnectionState.

• Start a timer:

  • Wait for Response datagram with matching CorrelationId.
  • If none comes within timeout → throw OperationCanceledException.

SendStreamingAsync:

• For this first iteration, throw NotSupportedException.
• Router should not route streaming endpoints over UDP; your routing config should enforce that.

SendCancelAsync:

• Optionally send a CANCEL datagram; but in practice, if requests are small, this is less useful. You can still implement it for symmetry.

5.4 Microservice: UDP connection

For microservice side:

• A single UdpClient bound to a local port.

• For each configured router (host/port):

  • HELLO: send a FrameType.Hello datagram.
  • HEARTBEAT: send periodic FrameType.Heartbeat.
  • REQUEST handling: not needed; UDP plugin is used for gateway → microservice only if you design it that way. More likely, microservice is the server in TCP, but for UDP you might decide microservice is listening on port and gateway sends requests. So invert roles if needed.

Given the complexity and limited utility, you can treat UDP as “advanced/optional transport” and implement it last.

---

6. Implement plugin 4: RabbitMQ

This is conceptually similar to what you had in Serdica.

6.1 Exchange/queue design

Decide and document (in Protocol & Transport Specification.md) something like:

• Exchange: stella.router

• Routing keys:

  • request.{serviceName}.{version} — gateway → microservice.
  • Microservice’s reply queue per instance: reply.{serviceName}.{version}.{instanceId}.

Rabbit usages:

• Gateway:

  • Publishes REQUEST frames to request.{serviceName}.{version}.
  • Consumes from reply.* for responses.

• Microservice:

  • Consumes from request.{serviceName}.{version}.
  • Publishes responses to its own reply queue; sets CorrelationId property.

6.2 Gateway: RabbitMqTransportClient

Implements ITransportClient:

• SendRequestAsync:

  • Create a message with:

    • Body = serialized Frame (REQUEST or buffered streaming).

    • Properties:

      • CorrelationId = frame.CorrelationId.
      • ReplyTo = microservice’s reply queue name for this instance.

  • Publish to request.{serviceName}.{version}.

  • Await a response:

    • Consumer on reply queue completes a TaskCompletionSource<Frame> keyed by correlation ID.

• SendStreamingAsync:

  • For v1, you can:

    • Only support buffered endpoints over RabbitMQ (like UDP).
    • Or send chunked messages (RequestStreamData frames as separate messages) and reconstruct on microservice side.

  • I’d recommend:

    • Start with buffered only over RabbitMQ.
    • Mark Rabbit as “no streaming support yet” in config.

• SendCancelAsync:

  • Option 1: send a separate CANCEL message with same CorrelationId.
  • Option 2: rely on timeout; cancellation doesn’t buy much given overhead.

6.3 Microservice: RabbitMQ listener

• Single IConnection and IModel.

• Declare and bind:

  • Service request queue: request.{serviceName}.{version}.
  • Reply queue: reply.{serviceName}.{version}.{instanceId}.

• Consume request queue:

  • On message:

    • Deserialize Frame.
    • Dispatch through IEndpointDispatcher.
    • Publish RESPONSE message to ReplyTo queue with same CorrelationId.

If you already have RabbitMQ experience from Serdica, this should feel familiar.

---

7. Routing config & transport selection

Projects: router config + microservice options Owner: config / platform dev

You need to define which transport is actually used in production.

7.1 Gateway config (RouterConfig)

Per service/instance, store:

• TransportType to listen on / expect connections for.
• Ports / Rabbit URLs / TLS settings.

Example shape in RouterConfig:

public sealed class ServiceInstanceConfig
{
    public string ServiceName { get; set; } = string.Empty;
    public string Version { get; set; } = string.Empty;
    public string Region { get; set; } = string.Empty;
    public TransportType TransportType { get; set; } = TransportType.Udp; // default
    public int Port { get; set; } // for TCP/UDP/TLS
    public string? RabbitConnectionString { get; set; }
    // TLS info, etc.
}

StellaOps.Gateway.WebService startup:

• Reads these configs.
• Starts corresponding ITransportServer instances.

7.2 Microservice options

StellaMicroserviceOptions.Routers entries must define:

• Host
• Port
• TransportType
• Any transport-specific settings (TLS, Rabbit URL).

At connect time, microservice chooses:

• For each RouterEndpointConfig, instantiate the right connector:

  switch(config.TransportType)
  {
      case TransportType.Tcp:
          use TcpMicroserviceConnector;
          break;
      case TransportType.Certificate:
          use TlsMicroserviceConnector;
          break;
      case TransportType.Udp:
          use UdpMicroserviceConnector;
          break;
      case TransportType.RabbitMq:
          use RabbitMqMicroserviceConnector;
          break;
  }
  

---

8. Implementation order & testing strategy

Owner: tech lead

Do NOT try to implement all at once. Suggested order:

1. TCP:

   • Reuse InMemory test suite:

     • HELLO + endpoint registration.
     • REQUEST → RESPONSE.
     • CANCEL.
     • Heartbeats.
     • (Optional) streaming as buffered stub for v1, then add genuine streaming.

2. Certificate/TLS:

   • Wrap TCP logic in TLS.

   • Same tests, plus:

     • Certificate validation.
     • Mutual TLS if required.

3. RabbitMQ:

   • Start with buffered-only endpoints.
   • Mirror existing InMemory/TCP tests where payloads are small.
   • Add tests for connection resilience (reconnect, etc.).

4. UDP:

   • Implement only for very small buffered requests; no streaming.

   • Add tests that verify:

     • HELLO + basic health.
     • REQUEST → RESPONSE with small payload.
     • Proper timeouts.

At each stage, tests for that plugin must reuse the same microservice and gateway code that worked with InMemory. Only the transport factories change.

---

9. Done criteria for “Implement real transport plugins one by one”

You can consider step 9 done when:

• There are concrete implementations of ITransportServer + ITransportClient for:

  • TCP
  • Certificate/TLS
  • UDP (buffered only)
  • RabbitMQ (buffered at minimum)

• Gateway startup:

  • Reads RouterConfig.
  • Starts appropriate transport servers per node/region.

• Microservice SDK:

  • Reads StellaMicroserviceOptions.Routers.
  • Connects to router nodes using the configured TransportType.
  • Uses the same HELLO/HEARTBEAT/REQUEST/RESPONSE/CANCEL semantics as InMemory.

• The same functional tests that passed for InMemory:

  • Now pass with TCP plugin.
  • At least a subset pass with TLS, Rabbit, and UDP, honoring their constraints (no streaming on UDP, etc.).

From there, you can move into hardening each plugin (reconnect, backoff, error handling) and documenting “which transport to use when” in your router docs.