Cookies with non-ASCII causing all endpoints to give status code 400

[RagingRudolf]

Hi everyone,

I'm working on a AspNetCore project and I have run into a odd problem.
When I send cookies to Kestrel where the value of a cookie contains "æ" (a danish character) then the request will return a status code 400 and never reach my controller (used debugger).

With debugging level "Error" nothing is printed to the Kestrel console but if I change debugging level to "Information" I get the following information exception messsage:

Connection id "0HKUIAF77PQFU" bad request data: "The input string contains non-ASCII or null characters."
Microsoft.AspNetCore.Server.Kestrel.BadHttpRequestException: The input string contains non-ASCII or null characters.
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Infrastructure.MemoryPoolIteratorExtensions.GetAsciiString(MemoryPoolIterator start, MemoryPoolIterator end)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame.TakeMessageHeaders(SocketInput input, FrameRequestHeaders requestHeaders)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.<RequestProcessingAsync>d__2.MoveNext()
warn: Microsoft.AspNetCore.Server.Kestrel[0]
      Connection processing ended abnormally
Microsoft.AspNetCore.Server.Kestrel.BadHttpRequestException: The input string contains non-ASCII or null characters.
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Infrastructure.MemoryPoolIteratorExtensions.GetAsciiString(MemoryPoolIterator start, MemoryPoolIterator end)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame.TakeMessageHeaders(SocketInput input, FrameRequestHeaders requestHeaders)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.<RequestProcessingAsync>d__2.MoveNext()

The main problem is that the project we are working on is original a .NET 4.5 project so we can't just use "ae" instead (The cookie is set with clientside tracking).

If you need some more information to reproduce this just let me know.

Best regards Martin

[Tratcher]

The message is accurate, HTTP does not allow non-ascii in header values. Enforcement of this varies somewhat from server-to-server.

For reference, what's the raw byte sequence on the wire in this case? ANSI? UTF-8?

[RagingRudolf]

I have used Fiddler to get the raw http request

GET http://localhost:5050/pos HTTP/1.1
Host: localhost:5050
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch
Accept-Language: da-DK,da;q=0.8,en-US;q=0.6,en;q=0.4
Cookie: testCookie=Æ

I'm not sure where to specific find the byte sequence with Fiddler, or if I should use Wireshark instead?

[davidfowl]

Unfortunately there's no workaround for this as we expect headers to be ASCII like @Tratcher said.

[OzieGamma]

@RagingRudolf cant you base64 instead?

[RagingRudolf]

@OzieGamma unfortunately not.

I will figure out what we will do. My best guess right now is that we will convert danish characters into the ascii equivalents.

I understand the reason for why you throw an exception but would it make sense to allow exactly that error "bubble up" as an error in the logs? I used quite some time to figure out why the server just gave an error 400 (and we have been using the IIS before without any problems). It's almost by accident I figured out that the error only showed up in with logging level "Information".

[halter73]

The "information" level logging is a tough issue. Previously we have logged some bad requests using the "warning" or worse "error" log level.

There we some complaints, since developers don't always have full control the clients sending the bad requests. We settled on the information level to avoid noisy logs. In fact, in 1.1, there won't even be the "Connection processing ended abnormally" warning logged due to this change. The more detailed BadHttpRequestException will still be logged as information, and only logged once.

[benaadams]

A cookie value is a http header value which RFC 7230 Field Value Components specifies:

Most HTTP header field values are defined using common syntax components (token, quoted-string, and comment) separated by whitespace or specific delimiting characters. Delimiters are chosen from the set of US-ASCII visual characters not allowed in a token (DQUOTE and "(),/:;<=>?@[]{}").

With the allowed characters in the value (token) being

"!" / "#" / "$" / "%" / "&" / "'" / "*"
/ "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
/ DIGIT (0-9) / ALPHA (A-Z, a-z)

So non ascii characters in the cookie are invalid.

The usual approach is to URL encode or Base64 encode the value when setting it and then decode it when retrieving it.

[muratg]

What @benaadams said :)