keep local dns in resolv.conf when host network enabled

[crazy-max]

replaces and closes #3244
fixes #3210
related to docker/buildx#175 (comment)

[tonistiigi]

^ updates needed per comments above

[tonistiigi]

^

[TBBle]

So late to the party here, but after some testing with buildx, I'm wondering if meta.NetMode == pb.NetMode_HOST is the correct condition here, since if the worker is in host-network mode (as seen in the buildx docker-container driver), then its pb.NetMode_UNSET network provider will also be host-mode, but that (a) already didn't require --allow-insecure-entitlement network.host, and (b) now also won't get the host-copied resolv.conf.

In this case, the custom network is reachable by IP, so I am assume that we are truely using host-mode for those RUN commands, but fail to recognise that at this point in the code. Up until this PR, the difference was only cosmetic (absence of a "enabling HostNetworking" log message when meta.NetMode == pb.NetMode_UNSET, but still using the host provider in that case). (Although it's a bit odd that if you don't pass --allow-insecure-entitlement network.host, then simply by going with the defaults you get to use host-networking, but not if you're explicit).

Basically, this fix only works for the buildx docker-container driver case if you also pass --network host to the build, and that's surprising to me.

Although I see now that was explicitly dismissed in #4524 (comment) so maybe the issue needs to be addressed on the buildx side, e.g., when a custom network is specified for a docker-container builder, it automatically allows the entitlement and inserts --network host into each build call for that builder. (Maybe it should always do that, since in that code the user doesn't configure the buildkit networking behaviour, they configure the container's network behaviour, and rely on buildkit always running in host mode, when it matters. Although custom buildkitd.toml messes with that, as seen in the CNI example in the docs.)

Or would a worker's explicit network mode carry through into the RUN commands, and the problem is that the worker is being left to default to auto (currently host) in the first place. (Although I didn't notice anywhere that stores the worker's specified network mode, we only seem to keep the resulting provider map...)

[tonistiigi]

I'm wondering if meta.NetMode == pb.NetMode_HOST is the correct condition here, since if the worker is in host-network mode (as seen in the buildx docker-container driver), then its pb.NetMode_UNSET network provider will also be host-mode,

This is intentional. See #4524 (review) When network mode is unset then user should not make expectations about host DNS or interfaces being accessible by the container. Only thing they can expect is access to internet. Eg. we are changing the default to bridge #4352 in the future release for better security and to avoid collisions between containers.

it automatically allows the entitlement and inserts --network host into each build call for that builder.

Indeed, in a container context, I think we could allow daemon-side entitlement automatically because "host" there already means container-host, not machine-host. But no --network host automatically as that would change the default execution context. If users used docker/buildx#175 for custom libnetwork networks in legacy builder then they already passed --network so setting --network host in the build should not add any additional complexity.

[TBBle]

I'm wondering if meta.NetMode == pb.NetMode_HOST is the correct condition here, since if the worker is in host-network mode (as seen in the buildx docker-container driver), then its pb.NetMode_UNSET network provider will also be host-mode,

This is intentional. See #4524 (review) When network mode is unset then user should not make expectations about host DNS or interfaces being accessible by the container. Only thing they can expect is access to internet. Eg. we are changing the default to bridge #4352 in the future release for better security and to avoid collisions between containers.

Yeah, I can see the logic of this on the BuildKit side. For buildx, the user is making an explicit request that the custom network be available to the build, so I guess that makes it a buildx responsibility.

Perhaps buildx could actually hook the custom network up to a bridge, and just pass the bridge name in, taking advantage of #4352, and avoiding the "host-networking" in RUN and the discrepancy between RUN and COPY.

it automatically allows the entitlement and inserts --network host into each build call for that builder.

Indeed, in a container context, I think we could allow daemon-side entitlement automatically because "host" there already means container-host, not machine-host. But no --network host automatically as that would change the default execution context. If users used docker/buildx#175 for custom libnetwork networks in legacy builder then they already passed --network so setting --network host in the build should not add any additional complexity.

I think this adds mild additional complexity simply because for a user building against a custom network, --network host is a semantic mismatch, it only happens to be "host" because the builder was set up inside a container. It just feels fiddly to me that if you've already created the builder on a custom network, then even supporting (and validating/enforcing) --network custom-network-name feels slightly repetitive. But I guess there are use-cases where you want ADD/COPY to see the custom network, but RUN to not see it...

Anyway, this sounds like a buildx thing too, so I guess that answers my questions as far as this PR is concerned. Thank you.

[penenkel]

@crazy-max I have a question regarding this change: Where does the file resolv-host.conf come from? Is that a file that is provided automatically (if so which component is responsible for that?) or do I have to ensure its existence myself?

[TBBle]

As far as I can see, it should be generated the first time through GetResolvConf, and any other time it needs to be updated, when running in (explicit) host-network mode.

[crazy-max]

@crazy-max I have a question regarding this change: Where does the file resolv-host.conf come from? Is that a file that is provided automatically (if so which component is responsible for that?) or do I have to ensure its existence myself?

It's not user-facing, just internal to handle cache for resolve conf in state dir.

[penenkel]

Ah, right, I misread the code. Thanks