concurrent requests cause StreamIDTooSmall error

[i9]

grpc-swift version 1.0.0-alpha.17, swift-nio-http2 from: "1.12.1"

grpc-swift client(iOS App) --- nginx --- grpc server

client sends 3 different rpcs almost at the same time, one failed due to StreamIDTooSmall, the other two failed due to unavailable. error log:

2020-07-27T15:19:49+0800 error: connection_id=4003FC27-24A7-4B11-B041-37FDCA5E35C1/0 error=StreamIDTooSmall() grpc client error
2020-07-27T15:19:49+0800 info: new_state=transientFailure old_state=ready connection_id=4003FC27-24A7-4B11-B041-37FDCA5E35C1/0 connectivity state change
"gRPC state did change from ready to transientFailure"
2020-07-27T15:19:49+0800 error: request_id=9168A18B-3ED8-43F2-BCF5-188C1F68FF9A call_state=active connection_id=4003FC27-24A7-4B11-B041-37FDCA5E35C1/0 error=unavailable (14) grpc client error
2020-07-27T15:19:49+0800 error: call_state=active request_id=517D243A-8A9B-413C-97E9-7078E3866614 connection_id=4003FC27-24A7-4B11-B041-37FDCA5E35C1/0 error=unavailable (14) grpc client error

But from nginx log, 2 out of 3 responses are returned successfully. one had error due to client prematurely closed connection while processing HTTP/2 connection which we assume is due to client side close and reconnect.

we suspect in createStreamChannel, self.nextOutboundStreamID = HTTP2StreamID(Int32(streamID) + 2) isn't thread safe, but we couldn't find any doc or guidance what to do.

[Lukasa]

nextOutboundStreamID is definitely thread-safe, as we guard it by an eventLoop.execute which forces a swap onto the event loop thread.

The issue here is actually long-standing:swift-nio-http2 allocates stream IDs before it has actually created the stream. It does this for a number of reasons, but it is fundamentally brittle. The result is that if you are calling createStreamChannel repeatedly and quickly you need to ensure that you actually write data on those channels in the order that you created them, or you'll see this error, where the stream channel created second sends its data first.

There are a bunch of things we can do to fix this, but all of them are quite substantial API changes. The ultimate behaviour has to be to create a new HTTP stream channel that does not expect its users to know the stream ID (that is, where the data being passed in the pipeline is HTTP2Frame.Payload, not HTTP2Frame). Then we can lazily allocate the stream ID as and when the channel sends its first payload data.

This is not the kind of thing we'd be able to fix quickly, though, so we should also investigate a quick workaround in grpc.

[i9]

Thanks for the prompt response! will close this and continue discussions in grpc-swift repo.

[Lukasa]

Ok, tracking the underlying issue in #214.