Caddy on_demand_tls asking ask endpoint for self IP when no FQDN/SNI is used

[captainfalcon23]

Hello,

I've gone through the Caddy docs and not sure if this is expected behaviour. I have the following fairly simple setup, and Caddy is configured to do on_demand_tls and I have configured an ask endpoint which I control.

AWS NLB -> Caddy -> App

When I query the NLB using the FQDN (either it's default or a custom set one) I see the request is made to my ask endpoint just fine.

However, when a request is made to one of the public IPs of the NLB, the request made to the ask endpoint is actually the address of Caddy itself! Example log from my ask endpoint:

[03/May/2024:06:26:16 +0000] "GET /caddy?domain=10.0.0.62 HTTP/1.1" 404 - "-" "Go-http-client/2.0"

10.0.0.62 = the internal IP of my Caddy.

I have proved this by simply curling my NLB public IPs e.g. (curl https://1.2.3.4) and watching the ask endpoint logs.

I don't understand why this happens - I would imagine if there is no SNI, there should be no "ask" at all? Or at least, it should be asking for the IP queried, not the internal Caddy IP. Ideally, this shouldn't happen and Caddy should just reject the request.

The reason why this is an issue is, anyone on the internet can do a port scan of the NLB and force Caddy to make a request to my ask endpoint. This shouldn't be the case, and Caddy should only respond to FQDN's, in which it can ask the ask endpoint.

Here is a redacted Caddyfile:

{
        http_port 80
        https_port 443
        admin 127.0.0.1:8080
        servers :80 {
                name http
                protocols h1 h2
        }
        servers :443 {
                name https
                protocols h1 h2
        }
        servers :8443 {
                name https-static
                protocols h1 h2
        }
        log {
                output stdout
                level DEBUG
                format json {
                        time_format rfc3339
                }
        }
        on_demand_tls {
                ask https://app-backend.blah/caddy
        }
        storage s3 {
                host "s3.ap-southeast-2.amazonaws.com"
                bucket "XXXXXXXXXXXXXXXXXXXXXXXX"
                use_iam_provider true
                prefix "tls-certificates"
                insecure false
        }
}
http:// {
        header {
                Server "XXXXXXXXXX (http to https)"
        }
        redir / https://{host}{uri}
        log http-https-redirect {
                output stdout
                format json {
                        time_format rfc3339
                }
        }
}
:443 {
        header {
                Server "XXXXXXXXXXXXXX"
                x-X-host {host}
        }
        reverse_proxy {
                dynamic srv {
                        name _https._tcp.XXXXXXXXXXXXXXXXXXXX
                        refresh 1m
                }
                header_down -Server
                header_up Host app-frontend.XXXXXXXXXXXXXXXX
                header_up X-Real-IP {remote}
                header_up X-Forwarded-Server {host}
                header_up X-Forwarded-Port {port}
                header_up x-ca-requested-host {host}
                header_up webapp {http.request.header.Cookie}

                transport http {
                        tls
                        tls_server_name app-frontend.XXXXXXXXXXXXXXXX
                        tls_insecure_skip_verify
                }
        }
        tls XXXXXXXXXXXXXXXXX {
                on_demand
        }
        log main {
                output stdout
                format json {
                        time_format rfc3339
                }
        }
}
:8443 {
        header {
                Server "XXXXXXXXXX (static site)"
        }
        route /* {
                s3proxy {
                        bucket XXXXXXXXXXXXXX
                        region ap-southeast-2
                        index index.html
                        root /
                        endpoint s3.ap-southeast-2.amazonaws.com
                        errors /index.html
                }
        }
        tls /etc/caddy/static-site.crt /etc/caddy/static-site.key
        log static-s3 {
                output stdout
                format json {
                        time_format rfc3339
                }
        }
}

[captainfalcon23]

Potentially related:

caddyserver/certmagic#22
#2438

[francislavoie]

The reason why this is an issue is, anyone on the internet can do a port scan of the NLB and force Caddy to make a request to my ask endpoint. This shouldn't be the case, and Caddy should only respond to FQDN's, in which it can ask the ask endpoint.

I don't see that as an issue. Your ask endpoint must be fast (sub millisecond ideally), so it should reject patterns you don't like immediately.

There are legitimate usecases where IP certs are desirable, like intranet setups, where having On-Demand IP certs is useful.

As of v2.8.0-beta.1, you could write your own permission module (plugin) instead of using an ask endpoint if it would help you move your logic closer to Caddy to short-circuit faster if you prefer.

[captainfalcon23]

That makes sense, but regardless, why is the internal/private address of caddy being used in the ask request, when the public ip of the nlb was hit? This seems like a bug, no?

[captainfalcon23]

Noting also, my issue isn’t to do with speed, but instead wasting an afternoon trying to figure out how external ips are querying my internal ask endpoint for a private IP.

So two issues/questions:

1. Is this a bug? Ie shouldn’t the real ip of the nlb be asked for
2. Is there a way to disable asking for certs for IPs as that doesn’t fit my use case

[francislavoie]

but regardless, why is the internal/private address of caddy being used in the ask request, when the public ip of the nlb was hit?

Because that's the only thing Caddy has to work with. The public IP isn't on the TCP packet that reaches Caddy.

[mholt]

This is intended behavior. SNI is not supposed to contain an IP address, but the server may very well be configured to serve an IP address (or any host! many instances are configured as wildcard listeners). As such, when the SNI is missing, it makes sense to assume the IP address.

and Caddy should only respond to FQDN's, in which it can ask the ask endpoint.

You can configure on-demand TLS to be used for only the domain names you specify. Whatever you can match in a TLS connection policy: https://caddyserver.com/docs/json/apps/http/servers/tls_connection_policies/ -- so that's currently SNI, ALPN, and remote IP.

why is the internal/private address of caddy being used in the ask request, when the public ip of the nlb was hit? This seems like a bug, no?

No, because the LB accesses Caddy using its internal IP.

Closing, as everything is working as intended; but feel free to continue discussion if needed.

[captainfalcon23]

Hmmm, I’m failing to understand how you could even have on demand IP certs - wouldn’t they be limited to the IP of Caddy? As SNI can’t contain a literal IP.

Either way, is there currently absolutely no way to just have caddy reject requests with no SNI? Or for its self ip?

Would something like this work (can I even dynamically get the caddy self ip?)? #2068 (comment)

[captainfalcon23]

@mholt thanks, I will check it out and see if I can reject my internal range

[mholt]

Hmmm, I’m failing to understand how you could even have on demand IP certs - wouldn’t they be limited to the IP of Caddy? As SNI can’t contain a literal IP.

SNI can contain whatever the client puts in it -- you can't trust it -- but it's not supposed to contain any IPs.

Either way, is there currently absolutely no way to just have caddy reject requests with no SNI? Or for its self ip?

Hmm, well requests haven't happened before the TLS handshake occurs so I'm not quite sure what you're asking here.

[captainfalcon23]

Hmm, well requests haven't happened before the TLS handshake occurs so I'm not quite sure what you're asking here.

I want to reject the request, without asking the ask endpoint.

[francislavoie]

Hmmm, I’m failing to understand how you could even have on demand IP certs - wouldn’t they be limited to the IP of Caddy? As SNI can’t contain a literal IP.

If you use tls internal, then Caddy issues certs using its internal issuer. And it can do this with on-demand enabled. Doing this, regardless of the IP used to reach Caddy, Caddy will issue a cert. So 127.0.0.1 for requests from localhost, 192.168.1.50 or whatever from any other device on your LAN, or whatever your x.x.x.x WAN IP is.

I want to reject the request, without asking the ask endpoint.

Then write a permission plugin, like I said earlier.

[mholt]

I want to reject the request, without asking the ask endpoint.

Seconding Francis here. I'm still a little confused because there is no request until after the TLS handshake is completed, and not having an 'ask' endpoint opens your server up for abuse.

[captainfalcon23]

Okay no worries, thanks for your help both. I’ll just leave it as it is.

All I was thinking was, right now, the session doesn’t get established because the ask endpoint returns a 404. I wanted to not even get as far as asking the ask endpoint (in my case, my ask endpoint is part of my backend service). So, rather than asking the endpoint and getting a 404 and then caddy denying the request, I thought it might be cleaner to just have caddy reject the request and not even make it to the ask end point. But it seems it probably doesn’t really matter. Apologies also if my terminology isn’t 100% correct.

[mholt]

Ok, so if I understand correctly, what you're actually asking for is for Caddy to completely reject the TLS handshake if SNI is empty?

[captainfalcon23]

Yep, that is correct.

[mholt]

@captainfalcon23 Gotcha... I went ahead and implemented a quick drop setting to the JSON config. See 8d7ac18

[captainfalcon23]

That's awesome @mholt !! Would this be implemented in the caddy config within the tls_connection_policies block like:

tls_connection_policies {
     drop true
}

[mholt]

There is no tls_connection_policies block in the Caddyfile (it's implicit) -- I haven't wired this up to the Caddyfile for now, I only had a couple of minutes, but you can use JSON in the meantime 👍