rc1 INCOMPLETE_CHUNKED_ENCODING when using nginx

[staff0rd]

Opened this here also, but I'm getting the feeling this more specific to kestrel rc1.

I have a docker container that has an aspnet5-rc1 application inside, listening on port 5000. If I make a request directly to that port, the response is 200 and quick.

If I make a request via nginx, then the response data comes back immediately (200), but does not terminate and rather about a minute later the response fails with; (in chrome) ERR_INCOMPLETE_CHUNKED_ENCODING.

Error does not occur when using nginx with aspnet-beta7. Headers for both aspnet-beta7 and aspnet-rc1 are the same;

HTTP/1.1 200 OK
Server: nginx/1.9.6
Date: Sun, 08 Nov 2015 04:38:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Frame-Options: SAMEORIGIN

[Tratcher]

I have a local repro that looks similar. If I throw an exception after the first response write then there is no chunked terminator and the connection is closed gracefully rather than reset.

            app.Run(async (context) =>
            {
                await context.Response.WriteAsync("hello world");
                throw new Exception("Test exception");
            });

The behavior is even stranger behind HttpPlatformHandler. It appears to suppress the chunking error and re-chunk the response including a terminator. Verify if it behaves correctly after kestrel changes to a reset.

[staff0rd]

@Tratcher any hints as to why I'm seeing this behind nginx reverse proxy but not when accessing it directly?

[Tratcher]

@staff0rd That's odd. In my case the reverse proxy (HttpPlatformHandler) hid the behavior by re-chunking the response.

You just posted the headers, did any of the body make it through? Or did you do an explicit Flush to send the headers before an exception occurred?

[staff0rd]

@Tratcher if you browse to http://test.atqu.in you can see the issue occur. If you browse to http://test.atqu.in:5000 you'll reach the same site directly (without reverse proxy) and you'll see the issue does not occur. As you'll see, the whole site is is delivered both times, but the connection stays open on the reverse-proxy for about a minute until it finally dies with INCOMPLETE_CHUNKED_ENCODING.

The code is here and is just a fresh MVC 6 beta8 project updated to rc1, so no extra flushes occur, nor are any exceptions thrown.

[Tratcher]

@staff0rd Let's re-test your scenario after we get the identified bug fixed.

[cesarblum]

The problem here is the same as what we're facing in #368: we're not sending an RST when we expect to (ConnectionControl.End(ProduceEndType.SocketDisconnect)).

Libuv currently does not expose the underlying socket enough for us to force an RST. Someone tried to submit a PR in the past but it seems like that never got in: joyent/libuv#498

Basically what needs to be done is to set SO_LINGER on the socket with a timeout of 0. That should force an RST.

[cesarblum]

Forget my previous comment 😄

nginx makes HTTP/1.0 requests when acting as a proxy. There is a known issue for Connection: close requests (which HTTP/1.0 requests implicitly are): #406

There's an easy fix for putting Kestrel behind an nginx reverse proxy: set proxy_http_version 1.1; to have nginx send HTTP/1.1 requests.

[staff0rd]

This is already configuring HTTP/1.1. Looking at the response headers of test.atqu.in also confirms that HTTP/1.1 is being used - same headers as in first post above.

[cesarblum]

Can you try a different version of nginx? I repro'd your issue on 1.8 and found that using that setting fixed the problem. Then I switched to 1.9.6 and on that specific version the requests seem to take a long time to be served (but I didn't see the original issue), even when running on localhost. I don't see that happening on 1.9.7 though.

[staff0rd]

Updated, same symptoms - 1.3min response time.

Response headers;

HTTP/1.1 200 OK
Server: nginx/1.9.7
Date: Thu, 19 Nov 2015 23:49:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive

[cesarblum]

I can hit that URL just fine. Wasn't that the URL where you were seeing the error?

[staff0rd]

Seriously? I just tried chrome + firefox + edge. Same symptoms on all - page immediately pops up, but loading doesn't stop for 1 min. However, the page imediately pops up and stops loading if I go around nginx. (http://test.atqu.in:5000)

[cesarblum]

Ohh, sorry, I didn't notice it was still loading 😞

What addresses are you binding to? I figured out why 1.9.6 was taking so long to forward the request to Kestrel. I was binding Kestrel to my IPv6 loopback interface only, but nginx was first trying to forward the request to the IPv4 interface. It took a minute to time out and try the IPv6 interface. Maybe something similar is happening in your setup?

[staff0rd]

Do you mean this or something on the nginx side?

[cesarblum]

Yep, that. Although you're already binding to an IPv4 interface... Can you try binding to localhost or * and see if that makes a difference?

[staff0rd]

Its hosted in a docker container so I don't think that binding to localhost will work. What's the syntax for *, http://*:5000?

[cesarblum]

That's correct.

[staff0rd]

Nah, same result;

Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING

[paralin]

@benaadams Any way a fix for this can be made for rc1? I don't think switching the entire stack to the AspCore dev versions is wise but I can apply a single patch to rc1.

[davidfowl]

See #341 (comment) for a workaround:

proxy_set_header Connection keep-alive;

[amcdnl]

@benaadams I'm experiencing similar to this in RC2. See: #636 (comment)

[cesarblum]

#636 #734 seem to be the same as this one (actually #636 is about BrowserSync). Let's center the discussion here, since this was the original issue.

@amcdnl Which version of nginx are you using, and on which OS?

[halter73]

This issue is related to Kestrel not sending the chunked response suffix. This was fixed by e4fd91b which closed this issue and #406.

Unless people are still seeing INCOMPLETE_CHUNKED_ENCODING in chrome after the nginx timeout, I would close this.

[amcdnl]

@CesarBS I updated the details in #636 to reflect your questions.

[mikeharder]

@amcdnl: Can we close this issue, or are you still seeing INCOMPLETE_CHUNKED_ENCODING with RC2?

[amcdnl]

@mikeharder Good to go.

[jmbalanag]

Im currently encoutering this issue:

"Microsoft.AspNetCore.Server.Kestrel": "1.0.0",

[davidfowl]

@jmbalanag can you provide more details about your setup and provide specific repro steps?

[jmbalanag]

This is the problem code:
[Route("api/lib_fund_source")]
public ActionResult GetFundSource()
{
return Json(db.lib_fund_source.Where(x => x.is_active == true).Select(x => new { Id = x.fund_source_id, Name = x.name }));

    }

if i remove, Where(x => x.is_active == true) it works just fine.

I think Its no longer related with kestrel.

[muratg]

@jmbalang Yeah, doesn't look like a Kestrel issue