git.stella-ops.org/docs/modules/router/messaging-valkey-transport.md

# Router · Messaging Transport over Valkey

## Status
- **Implemented** in Sprint 8100.0011.0003.
- Core components: Gateway DI wiring, GatewayHostedService integration, GatewayTransportClient dispatch.
- Last updated: 2026-04-05 (UTC).

## Purpose
Enable Gateway ↔ microservice Router traffic over an offline-friendly, Redis-compatible transport (Valkey) by using the existing **Messaging** transport layer:
- Router transport: `StellaOps.Router.Transport.Messaging`
- Messaging backend: `StellaOps.Messaging.Transport.Valkey`

This supports environments where direct TCP/TLS microservice connections are undesirable, and where an internal message bus is the preferred control plane.

## Implementation Summary

### Libraries
- `StellaOps.Router.Transport.Messaging` — Router transport layer over messaging
- `StellaOps.Messaging.Transport.Valkey` — Valkey/Redis backend for messaging
- `StellaOps.Messaging` — Core messaging abstractions and DI

### Gateway Integration
- **Program.cs** — Conditional registration of `ValkeyTransportPlugin` and `AddMessagingTransportServer()`
- **GatewayOptions.cs** — `GatewayMessagingTransportOptions` for Valkey connection and queue configuration
- **GatewayHostedService.cs** — Start/stop `MessagingTransportServer`, subscribe to events
- **GatewayTransportClient.cs** — Dispatch to `TransportType.Messaging` connections

## High-Level Flow
1) Microservice connects via messaging transport:
   - publishes a slim `HELLO` message with instance identity to the gateway control queue
2) Gateway processes HELLO:
   - registers the connection identity and requests endpoint metadata replay when needed
3) Microservice answers the replay request:
   - publishes an `EndpointsUpdate` frame with endpoints, schemas, and OpenAPI metadata
4) Gateway applies the metadata replay:
   - updates routing state, effective claims, and aggregated OpenAPI
5) Gateway routes an HTTP request to a microservice:
   - publishes a REQUEST message to the service request queue
6) Microservice handles request:
   - executes handler (or ASP.NET bridge) and publishes a RESPONSE message
7) Gateway returns response to the client.

Messaging-specific recovery behavior:
- Startup resync: the gateway sends `ResyncRequest` immediately after a slim `HELLO`.
- Administrative resync: `POST /api/v1/gateway/administration/router/resync` can request replay for one connection or the whole messaging fleet.
- Gateway-state miss: if a heartbeat arrives for an unknown messaging connection, the gateway seeds minimal state from the heartbeat identity and requests replay instead of waiting for a reconnect.

## Queue Topology (Conceptual)
The Messaging transport uses a small set of queues (names are configurable):
- **Gateway control queue**: receives service-to-gateway HELLO / HEARTBEAT / ENDPOINTS_UPDATE / RESPONSE frames
- **Per-service incoming queues**: gateway publishes REQUEST / CANCEL / RESYNC_REQUEST frames targeted to a service
- **Dead letter queues** (optional): for messages that exceed retries/leases

## Configuration

### Gateway YAML Configuration
```yaml
Gateway:
  Transports:
    Messaging:
      Enabled: true
      ConnectionString: "valkey:6379"
      Database: 0
      RequestQueueTemplate: "router:requests:{service}"
      ResponseQueueName: "router:responses"
      ConsumerGroup: "router-gateway"
      RequestTimeout: "30s"
      LeaseDuration: "5m"
      BatchSize: 10
      HeartbeatInterval: "10s"
```

### Gateway DI Registration
```csharp
// In Program.cs (already implemented)
if (bootstrapOptions.Transports.Messaging.Enabled)
{
    builder.Services.AddMessagingTransport<ValkeyTransportPlugin>(
        builder.Configuration, "Gateway:Transports:Messaging");
    builder.Services.AddMessagingTransportServer();
}
```

### Microservice
- Register router transports via plugin loading (no hard transport references in `StellaOps.Router.AspNet`).
- Use `AddRouterMicroservice(...)` from `StellaOps.Router.AspNet`; it resolves configured gateway transport types through `RouterTransportPluginLoader`.
- For messaging mode, the `StellaOps.Router.Transport.Messaging` plugin registers:
  - backend messaging plugin loading (`AddMessagingPlugins(...)`, env/config key `transport=valkey`)
  - Router messaging transport client (`AddMessagingTransportClient`)
- Ensure the following plugin DLLs are available either as service dependencies or under configured plugin directories:
  - `StellaOps.Router.Transport.Messaging.dll`
  - `StellaOps.Messaging.Transport.Valkey.dll`

## Operational Semantics (Draft)
- **At-least-once** delivery: message queues and leases imply retries are possible; handlers should be idempotent where feasible.
- **Lease timeouts**: must be tuned to max handler execution time; long-running tasks should respond with 202 + job id rather than blocking.
- **Determinism**: message ordering may vary; Router must not depend on arrival order for correctness (only for freshness/telemetry).
- **Push-first with recovery fallback**: Valkey Pub/Sub notifications wake consumers immediately when possible. If notifications silently stop, the queue layer still wakes via timeout fallback, connection-restored hooks, and randomized proactive re-subscription so requests and resync control frames do not wedge forever.
- **Queue fallback cost**: every wake can perform `XAUTOCLAIM` plus `XREADGROUP` checks before sleeping again. That traffic is expected resilience overhead, but it is materially smaller than replaying the full endpoint catalog on every heartbeat interval.

## Security Notes
- Messaging transport is internal. External identity must still be enforced at the Gateway.
- The Gateway must not trust client-supplied identity headers; it must overwrite reserved headers before dispatch.
  - See `docs/modules/gateway/identity-header-policy.md`.

## Implementation Status

### Completed (Sprint 8100.0011.0003)
1. ✅ Wire Messaging transport into Gateway:
    - start/stop `MessagingTransportServer` in `GatewayHostedService`
    - subscribe to `OnHelloReceived`, `OnHeartbeatReceived`, `OnEndpointsUpdated`, `OnResponseReceived`, `OnConnectionClosed` events
    - reuse routing state updates and claims store updates
2. ✅ Extend Gateway transport client to support `TransportType.Messaging` for dispatch.
3. ✅ Add config options (`GatewayMessagingTransportOptions`) and DI mappings.
4. ✅ Switch messaging registration from periodic full HELLO replay to explicit `ResyncRequest` / `EndpointsUpdate` control frames.

### Remaining Work
1. Add deployment examples (compose/helm) for Valkey transport.
2. Add integration tests using ValkeyFixture:
   - microservice HELLO registration via messaging
   - request dispatch + response return
3. Validate streaming support (or document as out-of-scope).