feature request: Standard way for microk8s to communicate about a local registry

[nicks]

Many Kubernetes clusters support a local registry, to make pushing images directly to the cluster fast. But every cluster documents it with manual shell scripts:

• https://kind.sigs.k8s.io/docs/user/local-registry/
• https://microk8s.io/docs/registry-built-in
• https://github.com/rancher/k3d/blob/master/docs/registries.md#using-a-local-registry
• https://docs.openshift.com/container-platform/4.3/registry/accessing-the-registry.html#registry-accessing-directly_accessing-the-registry

Because every cluster is slightly different, this makes it hard for tools to automate this work for users.

The big idea

Clusters should use annotations on the kube-system namespace to document how to configure a registry

Detailed design

I propose 3 new "standard" annotations:

x-k8s.io/registry: the host of the registry (hostname and port), as seen from outside of the cluster
x-k8s.io/registry-from-cluster: the host of the registry (hostname and port), as seen from inside the cluster
x-k8s.io/registry-help: The URL of documentation where users can read more about how to set a registry up.

Tools that would benefit from local registries should read this annotation and display it to the user

What this means for microk8s

All microk8s clusters would have a registry-help annotation pointing users to https://microk8s.io/docs/registry-built-in

When you run microk8s enable registry, microk8s would automatically add the x-k8s.io/registry annotation to nodes in the cluster.

Prior Art

This proposal is based on what we've been recommending to people in Tilt, with a lot of success: https://docs.tilt.dev/choosing_clusters.html#custom-clusters

update 5/4 - changed the proposal to use the kube-system namespace instead of node annotations

[nicks]

I also filed this feature request here kubernetes-sigs/kind#1543 and here k3d-io/k3d#234, and would love other ideas on places I should post about this

[nicks]

Just to keep you all in the loop, sig-cluster-lifecycle has suggested I open a KEP for this to kick the tires more on the proposal.

But I also recognize that microk8s falls slightly outside the KEP process. Are the microk8s maintainers ok with that?

[ktsakalozos]

Hi @nicks

Thank you for the suggestion and apologies for the late response.

I am +1 to the idea of formalizing on the registry use. I see your proposal this is under discussion. As soon as we have something concrete we can test and ship we will gladly do so.

Here is some info that you might find useful during your investigation. The registry add-on is enabled with microk8s enable registry. The registry manifest is available in [1] and the installation script is [2]. If you want to produce a demo/PoC to showcase your approach you can either build the snap following the instructions at [3] or just submit a PR and have the github action part of the PR checks produce a microk8s.snap for you.

Let me know if there is anything else we can do to assist you.

Thank you.

[1] https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/registry.yaml
[2] https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/enable.registry.sh
[3] https://github.com/ubuntu/microk8s/blob/master/docs/build.md

[nicks]

Thanks! That's helpful info. I'm hopeful this will be minimal engineering work to add, most of the work is driving consensus.

KEP PR: kubernetes/enhancements#1757
KEP tracking issue: kubernetes/enhancements#1755

[nicks]

Hi @ktsakalozos ! I'm doing a last call for any major objections or things I overlooked before we merge the initial spec

kubernetes/enhancements#1757
https://github.com/kubernetes/enhancements/blob/c5b6b632811c21ababa9e3565766b2d70614feec/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry/README.md#design-details

If you're OK with this proposal, a thumbs-up on the PR would be a help a lot! Thanks!

[ktsakalozos]

Looks good @nicks. You have my +1 on the PR.

[nicks]

The KEP has been marked implementable! I suspect it will be straightforward to implement, and I think I understand the microk8s internals well enough to send a PR if that would help, but it might take me a little while to get to it.