git.stella-ops.org/docs/modules/router/messaging-valkey-transport.md

Router · Messaging Transport over Valkey

Status

• Implemented in Sprint 8100.0011.0003.
• Core components: Gateway DI wiring, GatewayHostedService integration, GatewayTransportClient dispatch.
• Last updated: 2026-04-05 (UTC).

Purpose

Enable Gateway ↔ microservice Router traffic over an offline-friendly, Redis-compatible transport (Valkey) by using the existing Messaging transport layer:

• Router transport: StellaOps.Router.Transport.Messaging
• Messaging backend: StellaOps.Messaging.Transport.Valkey

This supports environments where direct TCP/TLS microservice connections are undesirable, and where an internal message bus is the preferred control plane.

Implementation Summary

Libraries

• StellaOps.Router.Transport.Messaging — Router transport layer over messaging
• StellaOps.Messaging.Transport.Valkey — Valkey/Redis backend for messaging
• StellaOps.Messaging — Core messaging abstractions and DI

Gateway Integration

• Program.cs — Conditional registration of ValkeyTransportPlugin and AddMessagingTransportServer()
• GatewayOptions.cs — GatewayMessagingTransportOptions for Valkey connection and queue configuration
• GatewayHostedService.cs — Start/stop MessagingTransportServer, subscribe to events
• GatewayTransportClient.cs — Dispatch to TransportType.Messaging connections

High-Level Flow

1. Microservice connects via messaging transport:
   • publishes a slim HELLO message with instance identity to the gateway control queue
2. Gateway processes HELLO:
   • registers the connection identity and requests endpoint metadata replay when needed
3. Microservice answers the replay request:
   • publishes an EndpointsUpdate frame with endpoints, schemas, and OpenAPI metadata
4. Gateway applies the metadata replay:
   • updates routing state, effective claims, and aggregated OpenAPI
5. Gateway routes an HTTP request to a microservice:
   • publishes a REQUEST message to the service request queue
6. Microservice handles request:
   • executes handler (or ASP.NET bridge) and publishes a RESPONSE message
7. Gateway returns response to the client.

Messaging-specific recovery behavior:

• Startup resync: the gateway sends ResyncRequest immediately after a slim HELLO.
• Administrative resync: POST /api/v1/gateway/administration/router/resync can request replay for one connection or the whole messaging fleet.
• Gateway-state miss: if a heartbeat arrives for an unknown messaging connection, the gateway seeds minimal state from the heartbeat identity and requests replay instead of waiting for a reconnect.

Queue Topology (Conceptual)

The Messaging transport uses a small set of queues (names are configurable):

• Gateway control queue: receives service-to-gateway HELLO / HEARTBEAT / ENDPOINTS_UPDATE / RESPONSE frames
• Per-service incoming queues: gateway publishes REQUEST / CANCEL / RESYNC_REQUEST frames targeted to a service
• Dead letter queues (optional): for messages that exceed retries/leases

Configuration

Gateway YAML Configuration

Gateway:
  Transports:
    Messaging:
      Enabled: true
      ConnectionString: "valkey:6379"
      Database: 0
      RequestQueueTemplate: "router:requests:{service}"
      ResponseQueueName: "router:responses"
      ConsumerGroup: "router-gateway"
      RequestTimeout: "30s"
      LeaseDuration: "5m"
      BatchSize: 10
      HeartbeatInterval: "10s"

Gateway DI Registration

// In Program.cs (already implemented)
if (bootstrapOptions.Transports.Messaging.Enabled)
{
    builder.Services.AddMessagingTransport<ValkeyTransportPlugin>(
        builder.Configuration, "Gateway:Transports:Messaging");
    builder.Services.AddMessagingTransportServer();
}

Microservice

• Register router transports via plugin loading (no hard transport references in StellaOps.Router.AspNet).
• Use AddRouterMicroservice(...) from StellaOps.Router.AspNet; it resolves configured gateway transport types through RouterTransportPluginLoader.
• For messaging mode, the StellaOps.Router.Transport.Messaging plugin registers:
  • backend messaging plugin loading (AddMessagingPlugins(...), env/config key transport=valkey)
  • Router messaging transport client (AddMessagingTransportClient)
• Ensure the following plugin DLLs are available either as service dependencies or under configured plugin directories:
  • StellaOps.Router.Transport.Messaging.dll
  • StellaOps.Messaging.Transport.Valkey.dll

Operational Semantics (Draft)

• At-least-once delivery: message queues and leases imply retries are possible; handlers should be idempotent where feasible.
• Lease timeouts: must be tuned to max handler execution time; long-running tasks should respond with 202 + job id rather than blocking.
• Determinism: message ordering may vary; Router must not depend on arrival order for correctness (only for freshness/telemetry).
• Push-first with recovery fallback: Valkey Pub/Sub notifications wake consumers immediately when possible. If notifications silently stop, the queue layer still wakes via timeout fallback, connection-restored hooks, and randomized proactive re-subscription so requests and resync control frames do not wedge forever.
• Queue fallback cost: every wake can perform XAUTOCLAIM plus XREADGROUP checks before sleeping again. That traffic is expected resilience overhead, but it is materially smaller than replaying the full endpoint catalog on every heartbeat interval.

Security Notes

• Messaging transport is internal. External identity must still be enforced at the Gateway.
• The Gateway must not trust client-supplied identity headers; it must overwrite reserved headers before dispatch.
  • See docs/modules/gateway/identity-header-policy.md.

Implementation Status

Completed (Sprint 8100.0011.0003)

1. ✅ Wire Messaging transport into Gateway:
   • start/stop MessagingTransportServer in GatewayHostedService
   • subscribe to OnHelloReceived, OnHeartbeatReceived, OnEndpointsUpdated, OnResponseReceived, OnConnectionClosed events
   • reuse routing state updates and claims store updates
2. ✅ Extend Gateway transport client to support TransportType.Messaging for dispatch.
3. ✅ Add config options (GatewayMessagingTransportOptions) and DI mappings.
4. ✅ Switch messaging registration from periodic full HELLO replay to explicit ResyncRequest / EndpointsUpdate control frames.

Remaining Work

1. Add deployment examples (compose/helm) for Valkey transport.
2. Add integration tests using ValkeyFixture:
   • microservice HELLO registration via messaging
   • request dispatch + response return
3. Validate streaming support (or document as out-of-scope).