H2 and H2C bi-direction streaming reverse proxy need non-nil req.body and flush headers before copyResponse

[masknu]

I found that haproxy is capable of proxying h2 clients to h2c upstream servers, but caddy's not working even with merged h2c branch #3289. For instance, client and server I used were v2ray(h2 outbound and h2c inbound).
After some digging and testing, I found the cause and have made caddy work in this scenario.

First, HTTP2 allows bi-direction streaming, and not using Transfer-Encoding and Content-Length headers for this. Thus we probably need to keep req.Body valid before invoking RoundTrip to upstream.

caddy/modules/caddyhttp/reverseproxy/reverseproxy.go

Line 424 in 7a99835

req.Body = nil // Issue golang/go#16036: nil Body for http.Transport retries

        req.Body = originBody
	start := time.Now()
	res, err := h.Transport.RoundTrip(req)

caddy/modules/caddyhttp/reverseproxy/reverseproxy.go

Line 500 in 7a99835

res, err := h.Transport.RoundTrip(req)

Second, in order to signal h2 clients begin sending streaming body, we need to flush response headers to client before copyResponse from res.Body to http.ResponseWriter. And use -1 as flushInterval for http2 connections.

	rw.WriteHeader(res.StatusCode)

	if h2 {
		if wf, ok := rw.(http.Flusher); ok {
			wf.Flush()
		}
		err = h.copyResponse(rw, res.Body, -1)
	} else {
		err = h.copyResponse(rw, res.Body, h.flushInterval(req, res))
	}

caddy/modules/caddyhttp/reverseproxy/reverseproxy.go

Line 605 in 7a99835

rw.WriteHeader(res.StatusCode)

The config file I used:

{
    "logging": {
        "logs": {
            "default": {
                "level": "DEBUG"
            }
        }
    },
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":443"
                    ],
                    "routes": [
                        {
                            "match": [
                                {
                                    "host": [
                                        "my.org"
                                    ]
                                }
                            ],
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "vars",
                                                    "root": "/usr/share/caddy"
                                                }
                                            ]
                                        },
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "headers": {
                                                        "request": {
                                                            "set": {
                                                                "Host": [
                                                                    "{http.request.host}"
                                                                ],
                                                                "X-Forwarded-For": [
                                                                    "{http.request.remote}"
                                                                ],
                                                                "X-Forwarded-Port": [
                                                                    "{server_port}"
                                                                ],
                                                                "X-Forwarded-Proto": [
                                                                    "http"
                                                                ],
                                                                "X-Real-Ip": [
                                                                    "{http.request.remote}"
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    "transport": {
                                                        "protocol": "http",
                                                        "compression": false,
                                                        "versions": [
                                                            "h2c",
                                                            "2"
                                                        ]
                                                    },
                                                    "upstreams": [
                                                        {
                                                            "dial": "localhost:54321"
                                                        }
                                                    ]
                                                }
                                            ],
                                            "match": [
                                                {
                                                    "path": [
                                                        "/toV2ray"
                                                    ]
                                                }
                                            ]
                                        },
                                        {
                                            "handle": [
                                                {
                                                    "handler": "file_server",
                                                    "hide": [
                                                        "/etc/caddy/Caddyfile"
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ]
                }
            }
        },
        "tls": {
            "automation": {
                "policies": [
                    {
                        "issuer": {
                            "email": "[email protected]",
                            "module": "acme"
                        }
                    }
                ]
            }
        }
    }
}

[mholt]

Interesting -- can you explain a little more? Is this a bug report or feature request? And did you make it work for you by removing the req.Body = nil line? What was the whole, resulting diff in the code? How can we reproduce the problem and test the patch? Thanks.

[masknu]

Yes, this is a bug report,. Caddy is unable to do h2 streaming for apps which need to get a header response before streaming content. And HAProxy is able to do that out of box.
I can send a PR to show the diff I made.
I didn't remove the req.body=nil line. In stead, I save the body to the request context, and reassign it to the req before the RoundTrip call if the req's MajorProto is 2.

[mholt]

@masknu Okay, gotcha. Yes, if you could submit a patch then I will be happy to review it! 👍

[masknu]

Sorry about the req.body part, after second confirm, it's not an issue.
The only issue is we need to flush the res.header under certain condition(proto==2 and contentlength == -1).

[masknu]

To reproduce the problem:

• Prepare local machine C, and an internet accessible server S.

• Copy server-config.json to S, replace "YOUR.DOMAIN" by your domain.

• Copy client-config.json to C, replace "YOUR.DOMAIN" by your domain.

• Copy caddy.json to S, replace "YOUR.DOMAIN" by your domain, replace "[email protected]" by your email address.

• Download v2ray (version 4.25.1) from v2ray and put it on C and S.

• Start v2ray at S with server-config.json by command:

  ./v2ray -config server-config.json

• Start caddy with caddy.json at step 4.

• Start v2ray at C with server-config.json by command:

  ./v2ray -config client-config.json

• Now, if Caddy behaves like HAProxy does with same configuration, we should be able to wget any internet URL at C by command:

  http_proxy=http://localhost:8080 https_proxy=http://localhost:8080 wget https://github.com/caddyserver/caddy/releases/download/v2.1.1/caddy_2.1.1_windows_amd64.zip

Below are the config files mentioned:

server-config.json:

{
    "log": {
        "loglevel": "debug"
    },
    "inbounds": [
        {
            "port": 54321,
            "listen": "127.0.0.1",
            "protocol": "vmess",
            "settings": {
                "clients": [
                    {
                        "id": "4c7c55bc-5d50-41cc-8f32-a5d205770524",
                        "level": 1,
                        "alterId": 64
                    }
                ]
            },
            "streamSettings": {
                "network": "h2",
                "security": "none",
                "httpSettings": {
                    "host": [
                        "YOUR.DOMAIN"
                    ],
                    "path": "/tov2ray"
                }
            }
        }
    ],
  "outbound": {
    "protocol": "freedom",
    "settings": {}
  },
  "outboundDetour": [
    {
      "protocol": "blackhole",
      "settings": {},
      "tag": "blocked"
    }
  ],
  "routing": {
    "strategy": "rules",
    "settings": {
      "rules": [
        {
          "type": "field",
          "ip": ["geoip:private"],
          "outboundTag": "blocked"
        }
      ]
    }
  }
}

client-config.json:

{
    "log": {
        "loglevel": "debug"
    },
    "inbounds": [
        {
            "listen": "0.0.0.0",
            "protocol": "http",
            "port": 8080,
            "settings": {}
        }
    ],
    "outbounds": [
        {
            "tag": "proxy",
            "protocol": "vmess",
            "settings": {
                "vnext": [
                    {
                        "address": "YOUR.DOMAIN",
                        "port": 443,
                        "users": [
                            {
                                "id": "4c7c55bc-5d50-41cc-8f32-a5d205770524",
                                "alterId": 64
                            }
                        ]
                    }
                ]
            },
            "streamSettings": {
                "network": "h2",
                "security": "tls",
                "httpSettings": {
                    "host": [
                        "YOUR.DOMAIN"
                    ],
                    "path": "/tov2ray"
                },
                "sockopt": {
                    "mark": 255
                }
            }
        },
        {
            "tag": "direct",
            "protocol": "freedom",
            "settings": {
                "domainStrategy": "UseIP"
            },
            "streamSettings": {
                "sockopt": {
                    "mark": 255
                }
            }
        },
        {
            "tag": "block",
            "protocol": "blackhole",
            "settings": {
                "response": {
                    "type": "http"
                }
            }
        },
        {
            "tag": "dns-out",
            "protocol": "dns",
            "streamSettings": {
                "sockopt": {
                    "mark": 255
                }
            }
        }
    ],
    "dns": {
        "hosts": {},
        "servers": [
            {
                "address": "223.5.5.5",
                "port": 53,
                "domains": [
                    "geosite:cn",
                    "ntp.org",
                    "YOUR.DOMAIN"
                ]
            },
            "8.8.8.8",
            "1.1.1.1",
            "114.114.114.114"
        ]
    },
    "routing": {
        "domainStrategy": "IPOnDemand",
        "rules": [
            {
                "type": "field",
                "ip": [
                    "223.5.5.5",
                    "114.114.114.114"
                ],
                "outboundTag": "direct"
            },
            {
                "type": "field",
                "ip": [
                    "8.8.8.8",
                    "1.1.1.1"
                ],
                "outboundTag": "proxy"
            },
            {
                "type": "chinasites",
                "outboundTag": "direct"
            },
            {
                "type": "field",
                "ip": [
                    "0.0.0.0/8",
                    "10.0.0.0/8",
                    "100.64.0.0/10",
                    "127.0.0.0/8",
                    "169.254.0.0/16",
                    "172.16.0.0/12",
                    "192.0.0.0/24",
                    "192.0.2.0/24",
                    "192.168.0.0/16",
                    "198.18.0.0/15",
                    "198.51.100.0/24",
                    "203.0.113.0/24",
                    "::1/128",
                    "fc00::/7",
                    "fe80::/10"
                ],
                "outboundTag": "direct"
            },
            {
                "type": "field",
                "domain": [
                    "geosite:category-ads-all"
                ],
                "outboundTag": "block"
            },
            {
                "type": "field",
                "protocol": [
                    "bittorrent"
                ],
                "outboundTag": "direct"
            },
            {
                "type": "chinaip",
                "outboundTag": "direct"
            },
            {
                "type": "field",
                "ip": [
                    "geoip:private",
                    "geoip:cn"
                ],
                "outboundTag": "direct"
            }
        ]
    }
}

caddy.json:

{
    "logging": {
        "logs": {
            "default": {
                "level": "DEBUG"
            }
        }
    },
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":443"
                    ],
                    "routes": [
                        {
                            "match": [
                                {
                                    "host": [
                                        "YOUR.DOMAIN"
                                    ]
                                }
                            ],
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "vars",
                                                    "root": "/usr/share/caddy"
                                                }
                                            ]
                                        },
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "headers": {
                                                        "request": {
                                                            "set": {
                                                                "Host": [
                                                                    "{http.request.host}"
                                                                ],
                                                                "X-Forwarded-For": [
                                                                    "{http.request.remote}"
                                                                ],
                                                                "X-Forwarded-Port": [
                                                                    "{server_port}"
                                                                ],
                                                                "X-Forwarded-Proto": [
                                                                    "http"
                                                                ],
                                                                "X-Real-Ip": [
                                                                    "{http.request.remote}"
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    "transport": {
                                                        "protocol": "http",
                                                        "compression": false,
                                                        "versions": [
                                                            "h2c",
                                                            "2"
                                                        ]
                                                    },
                                                    "upstreams": [
                                                        {
                                                            "dial": "localhost:54321"
                                                        }
                                                    ]
                                                }
                                            ],
                                            "match": [
                                                {
                                                    "path": [
                                                        "/tov2ray"
                                                    ]
                                                }
                                            ]
                                        },
                                        {
                                            "handle": [
                                                {
                                                    "handler": "file_server",
                                                    "hide": [
                                                        "/etc/caddy/Caddyfile"
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ]
                }
            }
        },
        "tls": {
            "automation": {
                "policies": [
                    {
                        "issuer": {
                            "email": "[email protected]",
                            "module": "acme"
                        }
                    }
                ]
            }
        }
    }
}

[choicky]

@masknu Hello Kevin, does caddy v2 work with v2ray (h2c)? if yes, could you share your configurations? Thank you very much.

The following Caddyfile does NOT works for me.

    reverse_proxy /{H2PATH} {
        to localhost:{PORT}
        header_up Host {http.request.host}
        header_up X-Forwarded-For {http.request.remote}
        header_up X-Forwarded-Port {server_port}
        header_up X-Forwarded-Proto "http"
        header_up X-Real-Ip {http.request.remote}
        transport http {
#            compression false
            versions h2c 2
        }
    }

[masknu]

Yes, it works well.
Everything looks fine with your config. If I were you, I would prefix localhost:{PORT} with http:// and move it to the front:

    reverse_proxy /{H2PATH}  http://localhost:{PORT}{
        transport http {
            versions h2c 2
        }
    }

And the server-config.json mentioned above is pretty much what my v2ray server config file looks like. The important part is "streamSettings"."security":

"streamSettings": {
                "network": "h2",
                "security": "none",
                "httpSettings": {
                    "host": [
                        "YOUR.DOMAIN"
                    ],
                    "path": "/tov2ray"
                }
            }

[choicky]

@masknu Thanks for your reply.

However, my config still does not work.

Could you please send a copy of your config files to my email? It's [email protected] .

[masknu]

@choicky Here is a set of complete config files:
https://gist.github.com/hang333/047ecf4c8d7d2868f1ce142d713a3520
from v2ray/discussion#759 .

[choicky]

@masknu Thank you very much. It works for me.

By the way, the ws configure could be simplified as reverse_proxy /{wsPath} localhost:{Port} It seems that the @v2ray_websocket is not required.