Overwrite remote_ip with x-forwarded-for for ip_hash load-balancing

[shufps]

Hi,

I've been researching for a couple of hours and still didn't find out if Caddy supports this or not.

My Caddy is behind another Reverse-Proxy (because it can't do ip_hash load-balancing 😩 ) but I can't simply swap it because it would require a lot of work that is not doable in a short timeframe.

I tried Nginx before but it doesn't support active health checks (except if paying for Plus).

After researching a lot I found out that Caddy is lightweight, has an appealing config file format and supports ip_hash.

Now I'm struggeling with the last problem - maybe a show-stopper, idk - that the load-balancer should use the IP from X-Forwarded-For as remote-ip before it is hashed.

I tried to set the X-Real-IP header to the X-Forwarded-For IP but it doesm't seem to work.

Somewhere else I found that the Log always shows the incoming headers, therefore always shows the IP of my other proxy, but according to the log, it's always the same upstream server that is selected. Therefore I assume that it's not working.

Any ideas if Caddy can do this?

Thx in advance!

[francislavoie]

Currently ip_hash only uses the remote address.

With all the work we've done recently with trusted_proxies and built-in client_ip parsing (see #5104), it should be trivial to add another policy that can read from the client IP instead.

Coincidentally, I just opened a PR a few hours ago to add another new lb policy. I'll probably add it to this PR in the coming days. #5468

[francislavoie]

Alright, updated #5468 with the new client_ip_hash policy. Feel free to build from that PR to try it out 👍

[shufps]

Alright, updated #5468 with the new client_ip_hash policy. Feel free to build from that PR to try it out +1

Wow, thx! I'll test it right away 🥳

[shufps]

just in case you need a working Dockerfile (based on nonroot-image):

Dockerfile.tar.gz

[shufps]

Almost there ... It seems the version prior to 2.7.0 can't choose which header to use for ip_hash_client, unfortunately 🥺

But I can change it in the code for testing and use X-Forwarded-For or Cf-Connecting-Ip.

edit:

"Cf-Connecting-Ip":["2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx"],"X-Forwarded-For":["172.x.x.x"],

Mmmhmm, these are arrays 🤔

I'm trying this now:

// Select returns an available host, if any.
func (ClientIPHashSelection) Select(pool UpstreamPool, req *http.Request, _ http.ResponseWriter) *Upstream {
	var address string
	addresses := caddyhttp.GetVar(req.Context(), "Cf-Connecting-Ip")
	switch x := addresses.(type) {
	case []string:
		address = strings.Join(x, ",")
	case string:
		address = x
	}
	clientIP, _, err := net.SplitHostPort(address)
	if err != nil {
		clientIP = address // no port
	}
	return hostByHashing(pool, clientIP)
}

[francislavoie]

That's not the right way to use it. You're meant to configure both trusted_proxies and client_ip_headers in the server global options, which will correctly set the client IP var, then the client_ip_hash policy can be used.

[shufps]

Hmm, this was the first I tried because I read it in some other PR (the one saying that the options change with 2.7.0) as comment.

But:

Error: adapting config using caddyfile: /etc/caddy/Caddyfile:2: unrecognized global option: trusted_proxies

This is the commit I'm building:

commit 1785297f8134062d6ff5a779394914d862797520 (HEAD -> client_ip_hash, origin/query-lb-policy)
Author: Francis Lavoie <[email protected]>
Date:   Sun Apr 2 08:41:18 2023 -0400

    Change query policy to join all the values

And this the Global-Section of my Caddyfile:

{
        trusted_proxies 172.0.0.0/8
        client_ip_hash Cf-Connecting-Ip
        
        log {
                level DEBUG
                output file /data/log/caddy_test.log {
                        roll_size 10MiB
                        roll_keep 10
                        roll_keep_for 336h
                }
        }
}  

[francislavoie]

Those go within servers in global options. See the docs: https://caddyserver.com/docs/caddyfile/options#trusted-proxies

[shufps]

Oh, ok sorry 🙈

I can confirm that it works 👍

Thx a lot 🙌

[alwismt]

My caddy version is v2.6.4, and i tried to do that and getting that error when i try to reload caddy reload, would mind telling me what's wrong? (I wanna get real client ip for rate limit)

2023/04/25 15:05:25.419	INFO	using adjacent Caddyfile
Error: adapting config using caddyfile: parsing caddyfile tokens for 'servers': Caddyfile:5 - Error during parsing: unrecognized servers option 'client_ip_hash'

{
	servers {
		# all traffic come from cloudflare cdn
		trusted_proxies cloudflare
		client_ip_hash Cf-Connecting-Ip
	}
	order rate_limit first
}
sg4.sitefire.com {
	# forward request to golang backend
	reverse_proxy [::]:8080
	# limit reqs
	rate_limit {
		zone dynamic {
			# wanna add unique ip for per user
			key {client_ip_hash}
			events 100
			window 1m
		}
	}
	encode gzip
	# just for verify that client_ip_hash works or nor
	respond {client_ip_hash}
}

[francislavoie]

client_ip_hash is an lb_policy, it doesn't go in servers.

You're probably looking for client_ip_headers from #5104.

But also neither of those are in any released version yet. Check the dates on the issues/PRs you're referencing vs the release date of the version you're using. You can build from the master branch if you need this right away.

[alwismt]

Yeah i cam from #5104. So in there in order to work do i need to build it from master? Can't i use xcaddy?

xcaddy build [2.7.0] \
	--with github.com/mholt/caddy-ratelimit \
	--with github.com/WeidiDeng/caddy-cloudflare-ip

[francislavoie]

Yes, but 2.7.0 doesn't exist yet. So you need to build from master. See https://github.com/caddyserver/caddy/releases