Force using /etc/containerd/certs.d for registry config.

[Romain-Geissler-1A]

This is a breaking change, announced in release v0.20. See https://kind.sigs.k8s.io/docs/user/local-registry/ how to setup a local registry.

Note: users who used to patch the containerd config to set explicitly:

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

should now remove this patch as it is now kind's default configuration.

This is a more brutal change than #3574 in which it was suggested to go the hard way and switch the default now.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Romain-Geissler-1A
Once this PR has been reviewed and has the lgtm label, please assign aojea for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @Romain-Geissler-1A. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Romain-Geissler-1A]

Hi,

Since you have just released, is there any chance to consider something like this before the next v0.24.0 release ?

[BenTheElder]

I'm not sure I understand the motivation, versus using containerdConfigPatches for those willing and able to get ahead of this.

Otherwise, it seems like we'd be best to hold off on breaking changes until we're forced to ship [edit: containerd] 2.x?

[Romain-Geissler-1A]

The motivation is that I also maintain software on my own, and I find a bit strange that the default is the deprecated (but historical) behavior. In my own software when I introduce an breaking change, after a reasonable period of time I switch the default to the non-deprecated state, then eventually drop the old one. I initially tried to propose something less breaking, but then you suggested we should be more breaking, so here it is.

I maintain a kindest/node derivative image for a corporate environment, used by hundreds of developers. My policy is like Fedora: upstream first (when applicable). So I proposed this instead of having to maintain an internal patch on my side. Now if you prefer waiting for the upgrade to containerd 2.0, fine, I can decline this pull request.

[BenTheElder]

The motivation is that I also maintain software on my own, and I find a bit strange that the default is the deprecated (but historical) behavior. In my own software when I introduce an breaking change, after a reasonable period of time I switch the default to the non-deprecated state, then eventually drop the old one. I initially tried to propose something less breaking, but then you suggested we should be more breaking, so here it is.

I agree with this sentiment, I think the distinction here is that we're not introducing this breaking change, but rather we've been preemptively warning users that we'll be forced to at some point (since forking containerd is infeasible / unreasonable for us and switching runtimes would be even more breaking).

And further that it costs us ~nothing to maintain the current state, and the new API is more difficult for users to use.

I maintain a kindest/node derivative image for a corporate environment, used by hundreds of developers. My policy is like Fedora: upstream first (when applicable).

Appreciated, a fair warning that from our point of view while details of the kind image leak and we don't make breaking changes for fun we also generally only advertise that it can run Kubernetes at a given version and future images might look rather different in ways we don't reasonably expect users to depend on (versus configuring registry mirrors in the runtime, that is unfortunately not really covered by the Kubernetes API but particularly useful for kind clusters).

So I proposed this instead of having to maintain an internal patch on my side. Now if you prefer waiting for the upgrade to containerd 2.0, fine, I can decline this pull request.

I'm not actually sure what is preferable. I would've liked to give users a smoother transition, I'm also not sure how soon we will have to do this, and there are unfortunately a lot of users that just adopt kindest/node images across releases without seeing the release notes.

[Romain-Geissler-1A]

That's why I initially proposed a smoother version which would try to "infer" from the user provided files if we are in one case or another, but maybe this could be refined to look for hosts.toml file for example.

Note that I just re-checked the upstream containerd documentation, and if I understand this commit containerd/containerd@c514630 correctly, it seems there is yet another breaking change (version 2 --> 3 and a change of plugin name) in containerd 2.0. So in effect 3 possible configuration: historical one with < 2.0, the move the "new" one with < 2.0, and the "real new" one with >= 2.0

[BenTheElder]

Note that I just re-checked the upstream containerd documentation, and if I understand this commit containerd/containerd@c514630 correctly, it seems there is yet another breaking change (version 2 --> 3 and a change of plugin name) in containerd 2.0. So in effect 3 possible configuration: historical one with < 2.0, the move the "new" one with < 2.0, and the "real new" one with >= 2.0

containerd 2.x still supports v2 config, which we are continuing to use for now.

I think what we'll have to do is make the containerdConfigPatches aware of a version field similar to how the kubeadmConfigPatches work, so users can start writing version specific patches, but that one isn't strictly required yet.

We have however switched to containerd 2.x in the next release (... after sorting out some other issues), and therefore will have this config by default.

The registry script will need to continue to support old releases and images ideally though, but perhaps we can add a comment noting that this part is not required for newer releases.

Thanks for your time, I think we can move ahead with just the containerd 2.x migration and highlight this aspect again in the release notes. We've been warning users for some time to get ahead of this particular change.