Binary at start of cached response from ASP.NET server

[andrewdonkin]

When I first request a page out of an API on localhost, everything works as expected. The output from restclient-http-send-current-raw is:

HTTP/1.1 200 OK
Date: Thu, 12 May 2022 22:16:25 GMT
Content-Type: application/json; charset=utf-8
Server: Kestrel
Cache-Control: no-cache
Transfer-Encoding: chunked

{ "json": "looks good" }

The output from restclient-http-send-current is the same with the comments at the bottom, bookended with the request and the elapsed time.

If I resend within a couple of minutes the output is very different. The headers are 252 unprintable bytes, represented in the buffer in octal (starting with \344\241\224\345\221\220 and ending with \340\251\273), followed by the JSON payload minus its opening brace. It is followed up by a null, ^@.

If I give it a minute or two and reload, it's fine again. Same if I swap an IP number instead of localhost, or if I use a second invocation of Emacs - they are fine for the first request, then full of gunk for the next. It's as though an Emacs-local cache is getting in the way.

Github interprets the unprintable bytes as UTF-16 [update: so does Emacs, sometimes], so we're going to get glyphs here instead of the octal that Emacs shows me in the buffer. This is from restclient-http-send-current:

// GET https://localhost:8904/api/
// 䡔呐⼱⸱′〰⁏䬊䑡瑥㨠周甬‱㈠䵡礠㈰㈲′㌺ㄶ㨱㐠䝍吊䍯湴敮琭呹灥㨠慰灬楣慴楯港橳潮㬠捨慲獥琽畴昭ㄶ੓敲癥爺⁋敳瑲敬੃慣桥ⵃ潮瑲潬㨠湯ⵣ慣桥੔牡湳晥爭䕮捯摩湧㨠捨畮步搊੻"json":"looks good"}\0Request duration: 0.015914s

There are exactly 1.5 x the number of "bad" bytes as header characters and their values smacked of UTF-8 so I ran it through a converter on the web. Luckily I picked a broken one, because it spat out

TP1120O
ae h,1 a 022:75 M
otn-ye plcto/sn hre=t-6Sre:KsrlCceCnrl occeTase-noig hne
{

...which is not the strict UTF-8 decode of what I gave it. It is the low byte of each of the 84 16-bit characters converted from the 252 bytes of UTF-8. It is every other character from the HTTP headers. See how HTTP/1.1 becomes TP11.

The content-type must be application/json; charset=utf-16 because it's been thinned to otn-ye plcto/sn hre=t-6.

Something is saying that the headers are UTF-16-encoded. Something else is dutifully interpreting them that way so that, for example, the final space and open brace, which are 0x0a and 0x7b in ASCII, are mashed together into 0x0a7b. Then it UTF-8-encodes it into 0xe0a9bb and I get \340\251\273 in the buffer.

The payload of the response renders correctly, except all on one line because (I guess) the pretty-printer is thrown off by the gunge in front. So either the payload is in UTF-16 and is being re-encoded to UTF-8 correctly, or whatever is molesting the headers is
leaving the payload alone.

On to what could be causing this. When I use curl to get the same URL twice on the same TLS session, it reports the content-type charset coming back as utf-8 both times. Curl says it is using a keepalive. Each request reaches the server - it's not being handled by some caching reverse-proxy (because there isn't one, and because the server is logging the request). Plus, two Emacsen on the same box don't affect each other's responses. So I believe it is not server-side.

Is the package that Emacs restclient uses for HTTP doing something screwy? How would I investigate that?

Emacs 27 and 28 on Windows. Looks like a duplicate of #195, but I think I have a bit more analysis in here so I'll leave the issue open, until someone objects.

Sure would appreciate any help. Thanks in advance.