reverseproxy: Support HTTP/3 transport to backend

[mholt]

VERY EXPERIMENTAL. Enables HTTP/3 protocol from proxy to backend ("upstream").

I still don't fully understand why this would be needed/useful, but here is a quick stab attempt with the APIs available to us.

Enabling HTTP/3 necessarily disables other HTTP versions. (If you specify version "3" in your config, you cannot have any other versions.) We do not support protocol downgrade or negotiation like web browsers do. That would probably add considerable latency and complexity where you should already have control over the backends. Additionally, other HTTP transport options don't apply to the HTTP/3 round-tripper because it does not use the http.Transport type like the lower HTTP versions.

Example Caddyfile:

# for demonstration purposes, the backend will ONLY support HTTP/3,
# but of course it doesn't have to be this way
{
	servers :1235 {
		protocols h3
	}
}

# this runs the proxy on HTTP but you can use HTTPS in yours
:1234 {
	reverse_proxy localhost:1235 {
		transport http {
			tls
			versions 3
		}
	}
	log
}

# this is the HTTP/3 backend
localhost:1235 {
	log
	respond "Hello world!"
}

Then run curl -v "http://localhost:1234/" and you'll see the response on the front-end using HTTP/1.1, but the proxy used HTTP/3 to the backend.

In the backend (:1235) we log the request so you can verify that it shows "proto": "HTTP/3.0" for the proxied request. The frontend (:1234) will log "proto": "HTTP/1.1".

Anyway, this seems to work, but with the limitations I mentioned above. We have similar, albeit slightly less tight, restrictions on the "h2c" transport as well.

(@marten-seemann You might like to know we're trying this.)

Closes #5086

[marten-seemann]

Interesting! As far as I know, this is pretty uncommon, most CDNs don't use H3 for connecting to the backend.

The fact that this is H3-only shows that there'd be value in creating an HTTP client that properly does Happy Eyeballs v3 (Tracking issue: quic-go/quic-go#4336. Not sure though if this would necessarily live inside of quic-go, or maybe in a small repo on the side).

[francislavoie]

This also means tls must be enabled, right? I don't think it's valid to do HTTP over QUIC? I can't see how this would ever have a performance advantage over HTTP/1.1 in private networks if it had to add TLS overhead 🤔

Anyway I see no reason not to merge this as-is. If it seems to work, that's cool. Fun toy, I guess.

[mholt]

Yeah, if you take out tls it won't work.

[mholt]

Eh, this is a pretty low-friction change and shouldn't affect any existing configs since it requires an explicit version 3 be configured; so I might just merge this for the pre-release and we can see how it goes.

[Forza-tng]

Seems to work well. Thanks!

One use case is to run caddy on a cloud server and reverse proxy to a self hosted server. Using plain http wouldn't be secure, and h3 is quic"er than h2.

Of course, running plain http over a VPN between the two is another option, though could be more difficult to set up.

[mholt]

h3 is quic"er than h2.

Have you verified that this is the case for your site?

[Forza-tng]

h3 is quic"er than h2.

Have you verified that this is the case for your site?

I just did a quick test using h2load locally on the server with the reverse proxy. The server has a wireguard vpn link to another server with caddy running. The latency is ~9.5ms over wireguard and ~9.1ms without wireguard.

• Over wireguard and h2c: ~750 req/s
• Over wireguard and h3: ~700 req/s
• No wireguard and h3: ~800 req/s
  Did not test h2c without wireguard as it is not encrypted.

The h2load command was: h2load -n10000 -c10 --connect-to=127.0.0.0:443

[mholt]

Interesting; any idea what the h2 performance is while you're at it?

[Forza-tng]

Interesting; any idea what the h2 performance is while you're at it?

Forgot about that. h2 performed very similar to h3 with and without wireguard. h2c was slightly ahead over wireguard, while h2/h3 over plain internet was always faster than h2c over wireguard. The variation between runs over wireguard seemed larger than without.

All in all, maybe 'it depends' :)

One take away is still that Caddy is pretty fast how ever you see it. The reverse proxy caddy is running on a 4 core ARM Neoverse-N1 VM.

[mholt]

Great -- good to know, thanks for the numbers!