Add integration tests for migration categories and execution

- Implemented MigrationCategoryTests to validate migration categorization for startup, release, seed, and data migrations. - Added tests for edge cases, including null, empty, and whitespace migration names. - Created StartupMigrationHostTests to verify the behavior of the migration host with real PostgreSQL instances using Testcontainers. - Included tests for migration execution, schema creation, and handling of pending release migrations. - Added SQL migration files for testing: creating a test table, adding a column, a release migration, and seeding data.
2025-12-04 19:10:54 +02:00
parent 600f3a7a3c
commit 75f6942769
301 changed files with 32810 additions and 1128 deletions
--- a/docs/router/SPRINT_7000_0005_0004_streaming.md
+++ b/docs/router/SPRINT_7000_0005_0004_streaming.md
@@ -0,0 +1,215 @@
+# Sprint 7000-0005-0004 · Protocol Features · Streaming Support
+
+## Topic & Scope
+
+Implement streaming request/response support. Large payloads stream through the gateway as `REQUEST_STREAM_DATA` and `RESPONSE_STREAM_DATA` frames rather than being fully buffered.
+
+**Goal:** Enable large file uploads/downloads without memory exhaustion at gateway.
+
+**Working directories:**
+- `src/Gateway/StellaOps.Gateway.WebService/` (streaming dispatch)
+- `src/__Libraries/StellaOps.Microservice/` (streaming handlers)
+- `src/__Libraries/StellaOps.Router.Transport.InMemory/` (streaming frames)
+
+## Dependencies & Concurrency
+
+- **Upstream:** SPRINT_7000_0005_0003 (cancellation - streaming needs cancel support)
+- **Downstream:** SPRINT_7000_0005_0005 (payload limits)
+- **Parallel work:** None. Sequential.
+- **Cross-module impact:** SDK, Gateway, InMemory transport all modified.
+
+## Documentation Prerequisites
+
+- `docs/router/specs.md` (sections 5.4, 6.3, 7.5 - Streaming requirements)
+- `docs/router/08-Step.md` (streaming section)
+- `docs/router/implplan.md` (phase 8 guidance)
+
+> **BLOCKED Tasks:** Before working on BLOCKED tasks, review [../implplan/BLOCKED_DEPENDENCY_TREE.md](../implplan/BLOCKED_DEPENDENCY_TREE.md) for root blockers and dependencies.
+
+## Delivery Tracker
+
+| # | Task ID | Status | Description | Working Directory |
+|---|---------|--------|-------------|-------------------|
+| 1 | STR-001 | TODO | Add SupportsStreaming flag to EndpointDescriptor | Common |
+| 2 | STR-002 | TODO | Add streaming attribute support to [StellaEndpoint] | Common |
+| 3 | STR-010 | TODO | Implement REQUEST_STREAM_DATA frame handling in transport | InMemory |
+| 4 | STR-011 | TODO | Implement RESPONSE_STREAM_DATA frame handling in transport | InMemory |
+| 5 | STR-012 | TODO | Implement end-of-stream signaling | InMemory |
+| 6 | STR-020 | TODO | Implement streaming request dispatch in gateway | Gateway |
+| 7 | STR-021 | TODO | Pipe HTTP body stream → REQUEST_STREAM_DATA frames | Gateway |
+| 8 | STR-022 | TODO | Implement chunking for stream data | Configurable chunk size |
+| 9 | STR-023 | TODO | Honor cancellation during streaming | Gateway |
+| 10 | STR-030 | TODO | Implement streaming response handling in gateway | Gateway |
+| 11 | STR-031 | TODO | Pipe RESPONSE_STREAM_DATA frames → HTTP response | Gateway |
+| 12 | STR-032 | TODO | Set chunked transfer encoding | Gateway |
+| 13 | STR-040 | TODO | Implement streaming body in RawRequestContext | Microservice |
+| 14 | STR-041 | TODO | Expose Body as async-readable stream | Microservice |
+| 15 | STR-042 | TODO | Implement backpressure (slow consumer) | Microservice |
+| 16 | STR-050 | TODO | Implement streaming response writing | Microservice |
+| 17 | STR-051 | TODO | Expose WriteBodyAsync for streaming output | Microservice |
+| 18 | STR-052 | TODO | Chunk output into RESPONSE_STREAM_DATA frames | Microservice |
+| 19 | STR-060 | TODO | Implement IRawStellaEndpoint streaming pattern | Microservice |
+| 20 | STR-061 | TODO | Document streaming handler guidelines | Docs |
+| 21 | STR-070 | TODO | Write integration tests for upload streaming | |
+| 22 | STR-071 | TODO | Write integration tests for download streaming | |
+| 23 | STR-072 | TODO | Write tests for cancellation during streaming | |
+
+## Streaming Frame Protocol
+
+### Request Streaming
+```
+Gateway → Microservice:
+1. REQUEST frame (headers, method, path, CorrelationId)
+2. REQUEST_STREAM_DATA frame (chunk 1)
+3. REQUEST_STREAM_DATA frame (chunk 2)
+...
+N. REQUEST_STREAM_DATA frame (final chunk, EndOfStream=true)
+```
+
+### Response Streaming
+```
+Microservice → Gateway:
+1. RESPONSE frame (status code, headers, CorrelationId)
+2. RESPONSE_STREAM_DATA frame (chunk 1)
+3. RESPONSE_STREAM_DATA frame (chunk 2)
+...
+N. RESPONSE_STREAM_DATA frame (final chunk, EndOfStream=true)
+```
+
+## StreamDataPayload
+
+```csharp
+public sealed class StreamDataPayload
+{
+    public Guid CorrelationId { get; init; }
+    public byte[] Data { get; init; } = Array.Empty<byte>();
+    public bool EndOfStream { get; init; }
+    public int SequenceNumber { get; init; }
+}
+```
+
+## Gateway Streaming Dispatch
+
+```csharp
+// In TransportDispatchMiddleware
+if (endpoint.SupportsStreaming)
+{
+    await DispatchStreamingAsync(context, transport, decision, cancellationToken);
+}
+else
+{
+    await DispatchBufferedAsync(context, transport, decision, cancellationToken);
+}
+
+private async Task DispatchStreamingAsync(...)
+{
+    // Send REQUEST header
+    var requestFrame = BuildRequestHeaderFrame(context);
+    await transport.SendFrameAsync(connection, requestFrame, ct);
+
+    // Stream body chunks
+    var buffer = new byte[_options.StreamChunkSize];
+    int bytesRead;
+    int sequence = 0;
+
+    while ((bytesRead = await context.Request.Body.ReadAsync(buffer, ct)) > 0)
+    {
+        var streamFrame = new Frame
+        {
+            Type = FrameType.RequestStreamData,
+            CorrelationId = requestFrame.CorrelationId,
+            Payload = SerializeStreamData(buffer[..bytesRead], sequence++, endOfStream: false)
+        };
+        await transport.SendFrameAsync(connection, streamFrame, ct);
+    }
+
+    // Send end-of-stream
+    var endFrame = new Frame
+    {
+        Type = FrameType.RequestStreamData,
+        CorrelationId = requestFrame.CorrelationId,
+        Payload = SerializeStreamData(Array.Empty<byte>(), sequence, endOfStream: true)
+    };
+    await transport.SendFrameAsync(connection, endFrame, ct);
+
+    // Receive response (streaming or buffered)
+    await ReceiveResponseAsync(context, transport, connection, requestFrame.CorrelationId, ct);
+}
+```
+
+## Microservice Streaming Handler
+
+```csharp
+[StellaEndpoint("POST", "/files/upload", SupportsStreaming = true)]
+public class FileUploadEndpoint : IRawStellaEndpoint
+{
+    public async Task<RawResponse> HandleAsync(RawRequestContext context, CancellationToken ct)
+    {
+        // Body is a stream that reads from REQUEST_STREAM_DATA frames
+        var tempPath = Path.GetTempFileName();
+
+        await using var fileStream = File.Create(tempPath);
+        await context.Body.CopyToAsync(fileStream, ct);
+
+        return RawResponse.Ok($"Uploaded {fileStream.Length} bytes");
+    }
+}
+
+[StellaEndpoint("GET", "/files/{id}/download", SupportsStreaming = true)]
+public class FileDownloadEndpoint : IRawStellaEndpoint
+{
+    public async Task<RawResponse> HandleAsync(RawRequestContext context, CancellationToken ct)
+    {
+        var fileId = context.PathParameters["id"];
+        var filePath = _storage.GetPath(fileId);
+
+        // Return streaming response
+        return new RawResponse
+        {
+            StatusCode = 200,
+            Body = File.OpenRead(filePath), // Stream, not buffered
+            Headers = new HeaderCollection
+            {
+                ["Content-Type"] = "application/octet-stream"
+            }
+        };
+    }
+}
+```
+
+## StreamingOptions
+
+```csharp
+public sealed class StreamingOptions
+{
+    public int ChunkSize { get; set; } = 64 * 1024; // 64KB default
+    public int MaxConcurrentStreams { get; set; } = 100;
+    public TimeSpan StreamIdleTimeout { get; set; } = TimeSpan.FromMinutes(5);
+}
+```
+
+## Exit Criteria
+
+Before marking this sprint DONE:
+1. [ ] REQUEST_STREAM_DATA frames implemented in transport
+2. [ ] RESPONSE_STREAM_DATA frames implemented in transport
+3. [ ] Gateway streams request body to microservice
+4. [ ] Gateway streams response body to HTTP client
+5. [ ] SDK exposes streaming Body in RawRequestContext
+6. [ ] SDK can write streaming response
+7. [ ] Cancellation works during streaming
+8. [ ] Integration tests for upload and download streaming
+
+## Execution Log
+
+| Date (UTC) | Update | Owner |
+|------------|--------|-------|
+| | | |
+
+## Decisions & Risks
+
+- Default chunk size: 64KB (tunable)
+- End-of-stream is explicit frame, not connection close
+- Backpressure via channel capacity (bounded channels)
+- Idle timeout cancels stuck streams
+- Typed handlers don't support streaming (use IRawStellaEndpoint)