large_unary interop test timing out against most grpc clients

[jtattermusch]

https://source.cloud.google.com/results/invocations/83cbf9c2-cf9c-48e6-9865-74f60f96a421/targets/github%2Fgrpc%2Finterop_test/tests;group=tests;test=cloud_to_cloud:c%2B%2B:aspnetcore_server:custom_metadata:tls;row=1

Not sure yet what's going on, but it can be reproduced with instructions here
https://github.com/grpc/grpc-dotnet/tree/master/testassets/InteropTestsWebsite

[jtattermusch]

It looks like ReadSingleMessageAsync gets stuck forever when reading a large unary request.

[jtattermusch]

CC @JunTaoLuo

[jtattermusch]

What I'm seeing is that ReadAsync() reads 98304 bytes in the first invocation and never returns from the second invocation.

The message sent by the interop client is slightly bigger than 271828 bytes.
https://github.com/grpc/grpc/blob/master/doc/interop-test-descriptions.md#large_unary

[jtattermusch]

I bisected the breakage down to
851e231 (which is basically just upgrade to preview 3)

So there are two options:

• either the usage pattern HttpContext's PipeReader of preview3 has changed
• there's a bug in preview3 that breaks our code

851e23101dd472703fd7e6caa91aeffa13f041ec is the first bad commit
commit 851e23101dd472703fd7e6caa91aeffa13f041ec
Author: Jan Tattermusch <[email protected]>
Date:   Thu Mar 7 09:36:18 2019 +0100

    Use .NET Core 3 preview 3 (#146)
    
    * bump to .NET Core 3 preview 3
    
    * use Grpc 1.19.0 nugets
    
    * nits in test runner scripts

:040000 040000 1ceea3b717fa44ccdf2d95fda7a259dfcf40d0dd ee9ed0e3caf0f016f2a218fdc9ed1d3c2c2f8e9b M	build
:100755 100755 8ae1db70c98764cbd9d080abb72b127c194bcc8e dcb1403eed4f7e3401925becf1942436686c2a73 M	build_and_test.sh
:100644 100644 f660a138ee650ca393d9a591c37cc168de911ddc 433a84f36505e22f8a8c2468b9f1524ac7cf906c M	global.json
:040000 040000 f62ee5d71e071600534d87781347de92af56965f b84a25afb89d9887afa2b5397fd289d57a712e4c M	kokoro

[davidfowl]

cc @jkotalik, likely something to do with your changes.

[jkotalik]

dotnet/aspnetcore#8200 may fix it. Other than that, I'm not sure and would need a simple repro.

[jkotalik]

Seems like just calling ReadAsync(100kb) and then calling Read again should cause it to repro?

[JamesNK]

Ping us once dotnet/aspnetcore#8200 is merged and we'll update to use a nightly SDK build. We'll test whether it still effects us.

[JunTaoLuo]

I think this is actually the stream limit being hit. https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.core.http2limits.initialstreamwindowsize?view=aspnetcore-2.2#Microsoft_AspNetCore_Server_Kestrel_Core_Http2Limits_InitialStreamWindowSize.

I was able to create a repro locally with a 100kb message and "fix" the problem by setting InitialStreamWindowSize to int.MaxValue. @jtattermusch can you try that and see if the interop tests still fail? Note that you'll probably need to also set InitialConnectionWindowSize if you are going above 128kb

[JunTaoLuo]

To clarify dotnet/aspnetcore#8200 should fix the behaviour in the long term but we can try setting InitialStreamWindowSize and InitialConnectionWindowSize for now as a workaround.