Edge, when referer url contains Chinese characters, MVC will respond code 400

[HumJ0218]

Is this a Bug or Feature request?:

BUG

Steps to reproduce (preferably a link to a GitHub repo with a repro project):

\0. Use browser Edge

\1. goto a url whitch contains Chinese characters like "http://localhost:60000/测试"

\2. goto a url hosted by Core MVC like "http://localhost:60000/anyone" or load other resources in \1's page
-- Edge will encode the request header "Referer" with octal encoding like "http://localhost:60000/\123\234\345...." (i don't remember detail, but it just like this, i got it by wireshark)

\3. Core MVC will return a response with http code 400

\4. whatever it's stand-alone or hosted by iis

\7. when it's others website server, like ##asp.net 4 mvc, it will return code 200 just as usual

Description of the problem:

Maybe if Core MVC dosen't decode the octal-encoded header?

Version of Microsoft.AspNetCore.Mvc or Microsoft.AspNetCore.App or Microsoft.AspNetCore.All:

.Net Core 2.0
Microsoft.AspNetCore.All 2.0.9 --(it is the newest version in core 2.0)
Microsoft.NETCore.App 2.0.0

[mkArtakMSFT]

Thanks for contacting us, @HumJ0218.
@Tratcher, what are your thoughts regarding this?

[pranavkm]

@HumJ0218 can you set the log level to Trace and include it here? Would be useful to figure out what's returning the 400.

[Tratcher]

Kestrel will reject this. I've asked @JunTaoLuo to reproduce it and check the raw bytes.

[JunTaoLuo]

Kestrel rejects non-ascii headers, aspnet/KestrelHttpServer#1144. UTF-8 encoding was added in 2.2.0-preview2 but I still don't think this request will be accepted since it doesn't look like it's UTF-8. I'll see if I can reproduce the issue and see what encoding is being used.

[JunTaoLuo]

Looks like this is a dupe of aspnet/KestrelHttpServer#1144. The Referer value 妲己 is likely being encoded as UTF-8 but Wireshark is displaying the encoding in octal as \345\246\262\345\267\261. I thought this was an unknown encoding at first since the octal display format confused me.

@HumJ0218 Can you try 2.2.0-preview2 when it releases to see if this issue is resolved? We started to accept requests with UTF-8 encoded header values in this release.

[HumJ0218]

@JunTaoLuo Thank you
I found Chrome will encode non-ascii to ascii(but show original chars in address bar). Edge or IE will be not. And Wireshark will show it in octal

I will try it in 2.2.0-p2 ASAP
Thank you again!

[HumJ0218]

It looks like Edge will encode the address I request, but not encode it when in others header content
Chrome will encode at all time.
so is it a bug in Edge?

[JunTaoLuo]

I'm not sure if it would be a bug.

According to spec, query string should be percent encoded: https://tools.ietf.org/html/rfc3986#section-3.4 (pchar is defined at https://tools.ietf.org/html/rfc3986#section-3.3). I think this is the reason why Chrome and Edge will encode the path line. However, I don't think there's a specified defined client behaviour in terms of header values so it's unlikely that this is a "bug" but rather just different client behaviour and implementation.

[JunTaoLuo]

@HumJ0218 Also, I'm assuming by "encode" you mean percent encoding? From the repro you have shown, the header values are still encoded by Edge, but the encoding it's using is UTF-8 instead of percent encoding.

[HumJ0218]

This is a test

Look at the first request, Edge sent (WHAT?) instead of "测试"?!!!

http://localhost:5000/?test=��

It's NOT %percent% encoding😂 but DO split to UTF-8 bytes and transfer in ASCII？

I'm confused

[HumJ0218]

maybe it's Edge's bug not Server‘s

[JunTaoLuo]

�� looks like an issue with Edge's display encoding. If you encode 测试 into UTF-8 bytes \xE6\xB5\x8B\xE8\xAF\x95, but then decode those raw bytes as extended ASCII, you'll get ��. This is a case of Edge not using the UTF-8 encoding for display but using UTF-8 encoding to convert the request line for the request. The Referer header looks to be jumbled though, and that's likely an Edge issue, can you retry using Chrome or Firefox and see if you have the same issue?

[HumJ0218]

Chrome is all good
no code 400

[JunTaoLuo]

@HumJ0218 What version of Edge and Windows are you running on? I cannot reproduce this locally on my machine.

Notice how the query string of the request and the value in the referer header both show %E6%B5%8B%E8%AF%95

[HumJ0218]

Windows 10 pro 1803 x64 , version 17134.228 ( zh-CN )

Edge 42.17134.1.0
EdgeHTML 17.17134

User-Agent:
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134"

should I upgrade them ?

[HumJ0218]

Either I type http://localhost:5000/?test=测试 or http://localhost:5000/?test=%E6%B5%8B%E8%AF%95 in Edge, the program can receive the parameter test is 测试

If it's %E6%B5%8B%E8%AF%95, SVG file or other's response code is 200. But if it's 测试, response code is 400.

[JunTaoLuo]

I've tested this in a VM and I'm able to reproduce it on a Chinese version of Windows 10. I can confirm this is a duplicate of aspnet/KestrelHttpServer#1144 and using 2.2.0-preview2 that will be released tomorrow will fix the issues:

Note that there is a bug with how Edge on a Chinese Windows 10 installation presents query string and headers in the developer tools view but both are sent as valid UTF-8 to the server.

[HumJ0218]

Thank you for your time!!

[JunTaoLuo]

I'm going to close this issue for now since we have an understanding of the underlying issue and there's no actionable items on our side. Let us know if you encounter any other issues.

[ghulamhussain]

hi i am facing exactly the same issue only on ms edge. can you tell me how you fixed this issue?

[Tratcher]

@ghulamhussain upgrade to 2.2.0-preview2.