support imagePullPolicy when pulling kubeadm images

[sefm]

Is this a request for help?

Bug / Help and possibly implement a fix for the bug

similar closed issue : #34

Versions

kubeadm version (use kubeadm version):
kubeadm version = kubeadm_1.8.2-00

ubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:38:10Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:48:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: AWS in isolated vpc.
• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Kernel (e.g. uname -a):
  Linux ip-10-205-78-6 4.4.0-1039-aws kubeadm join usage summary is missing master args #48-Ubuntu SMP Wed Oct 11 15:15:01 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

kubeadm init doesn't support arg = imagePullPolicy=never or local. This impose a problem when using kubeadm with no internet access since, it defaults trying to pull images from gcr.

kubeadm errors:
command : kubeadm init --kubernetes-version v1.8.2 --pod-network-cidr=10.244.0.0/16

[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.

What i have done . Downloaded all google_containers related to kubernetes 1.8.2 and then scp to the host and used docker to load the containers . kubeadm default policy imagePullPolicy=always . Therefore it was failing

docker images
REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE
gcr.io/google_containers/kube-apiserver-amd64            v1.8.2              6278a1092d08        12 days ago         194 MB
gcr.io/google_containers/kube-controller-manager-amd64   v1.8.2              5eabb0eae58b        12 days ago         129.2 MB
gcr.io/google_containers/kube-scheduler-amd64            v1.8.2              b48970f8473e        12 days ago         54.9 MB
gcr.io/google_containers/kube-proxy-amd64                v1.8.2              88e2c85d3d02        12 days ago         93.13 MB
gcr.io/google_containers/k8s-dns-sidecar-amd64           1.14.4              38bac66034a6        4 months ago        41.82 MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64          1.14.4              a8e00546bcf3        4 months ago        49.39 MB
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     1.14.4              f7f45b9cb733        4 months ago        41.42 MB
gcr.io/google_containers/etcd-amd64                      3.0.17              243830dae7dd        8 months ago        168.9 MB
gcr.io/google_containers/pause-amd64                     3.0     

To work around it . I have added imagePullPolicy=never to manifests and used kubelet to start the the pods .

docker ps 
CONTAINER ID        IMAGE                                                                                                                            COMMAND                  CREATED             STATUS              PORTS               NAMES
3469f8d57f49        gcr.io/google_containers/kube-controller-manager-amd64@sha256:c2cd4acd4238b2f2526abf5ba546d4e6f4a46618ad5747a539e8a72c294a7482   "kube-controller-mana"   5 minutes ago       Up 5 minutes                            k8s_kube-controller-manager_kube-controller-manager-ip-10-205-78-6_kube-system_a2384e51c277f0dc61222c242361a42d_0
9233a987f5ac        gcr.io/google_containers/kube-apiserver-amd64@sha256:3e980f4b57292568ea8c87be462cf0583e40bbc2dbfff71d0d9e19beda3cb74b            "kube-apiserver --sec"   5 minutes ago       Up 5 minutes                            k8s_kube-apiserver_kube-apiserver-ip-10-205-78-6_kube-system_52115e8757d49c532b9f9253a995f4c6_0
50574a1badf2        gcr.io/google_containers/kube-scheduler-amd64@sha256:7c920b718509e8cf811c69178526d84ebfab2bdbb95949f6e82eb5233e7b5f0e            "kube-scheduler --kub"   5 minutes ago       Up 5 minutes                            k8s_kube-scheduler_kube-scheduler-ip-10-205-78-6_kube-system_c277372d3697e2f5d4038d02914e31d8_0
aa7eb7294220        gcr.io/google_containers/etcd-amd64@sha256:d83d3545e06fb035db8512e33bd44afb55dea007a3abd7b17742d3ac6d235940                      "etcd --listen-client"   5 minutes ago       Up 5 minutes                            k8s_etcd_etcd-ip-10-205-78-6_kube-system_07f2b34fc77ee86c936d12f9da37f985_0
5a7251468f60        gcr.io/google_containers/pause-amd64:3.0                                                                                         "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_kube-scheduler-ip-10-205-78-6_kube-system_c277372d3697e2f5d4038d02914e31d8_0
9ec41787b709        gcr.io/google_containers/pause-amd64:3.0                                                                                         "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_kube-controller-manager-ip-10-205-78-6_kube-system_a2384e51c277f0dc61222c242361a42d_0
7e8ceedab55c        gcr.io/google_containers/pause-amd64:3.0                                                                                         "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_kube-apiserver-ip-10-205-78-6_kube-system_52115e8757d49c532b9f9253a995f4c6_0
7d1ee4d69ca6        gcr.io/google_containers/pause-amd64:3.0                                                                                         "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_etcd-ip-10-205-78-6_kube-system_07f2b34fc77ee86c936d12f9da37f985_0

master was up after that

kubectl cluster-info
Kubernetes master is running at https://10.205.78.6:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

etcd responding

curl 127.0.0.1:2379/version
{"etcdserver":"3.0.17","etcdcluster":"3.0.0"}

API

curl https://10.205.78.6:6443/api/ -k
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "10.205.78.6:6443"
    }
  ]

What you expected to happen?

expected kueadm to work if container-images are present locally.

How to reproduce it (as minimally and precisely as possible)?

reproduce by following the steps above.

Anything else we need to know?

not sure if it would it make sense to add hook kubeadm to talk to a private registry instead !

[luxas]

kubeadm default policy imagePullPolicy=always

This is not the case. The default imagePullPolicy is IfNotPresent, so this should work.

Can you post the relevant manifest yamls and the kubelet log when it doesn't work for you?

[sefm]

@luxas : Thanks for your response . Correct . The acceptable image imagePullPolicy values = Always , IfNotPresent and Never . After digging more the issue seems not related to imagePullPolicy . The issue was with not advertising the api server address :

Once I added --apiserver-advertise-address=10.205.78.6 it worked fine

kubeadm init --kubernetes-version=v1.8.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.205.78.6

Your Kubernetes master has initialized successfully!

Closing the issue.

[neolit123]

re-opening as we hit the same problem recently.
kubeadm might have to support image pull policy for it's CP images on the API side.

cc @bart0sh

[rosti]

@neolit123 is this the same issue? Do we want imagePullPolicy specifically for the Never case or for Always?

[neolit123]

re-using this issue so that we don't open a new one. a bit messy, but renamed it with a better title at least.

we need to support different policies.

[BenTheElder]

The Never case is correct for kind FWIW, though I think we're going to just start skipping preflight entirely (we typically fail and ignore most of the checks anyhow... 🙃 e.g. swap).

[fabriziopandini]

@neolit123 is this still a problem?
AFAIK kubeadm works in air gapped environment if the image exists locally; users also have options to influence image name (via image repository/image tag fields)

As far as I understand the problem, image pull policy should not add much to what described above..

[BenTheElder]

It doesn't work when e.g. it's trying to pull an unnecessary pause image and failing 🙃

…

On Sun, Apr 26, 2020, 10:24 Fabrizio Pandini ***@***.***> wrote: @neolit123 <https://github.com/neolit123> is this still a problem? AFAIK kubeadm works in air gapped environment if the image exists locally; users also have options to influence image name (via image repository/image tag fields) As far as I understand the problem, image pull policy should not add much to what described above.. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#524 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADKY7PWRRNAAS5VCVM2TRORU4JANCNFSM4ECQSPSQ> .

[neolit123]

@fabriziopandini

@neolit123 is this still a problem?

it was recently requested by @ncdc for CAPI use too.
our kubeadm logic currently always does a "IfNotPresent":
https://github.com/kubernetes/kubernetes/blob/0acf2f0983d1491caf60367f12c1bd76651209cc/cmd/kubeadm/app/preflight/checks.go#L835-L851

and with a policy of "Never" it will never pull and "Always" would always pull, which will actually run faster than "IfNotPresent" if the image is present locally.

[ncdc]

@neolit123 I think you're referring to the regression in kubeadm that always pulled instead of doing IfNotPresent? I don't think CAPI needs anything now that the regression has been fixed.

[neolit123]

ok, so i had to refresh my memory about that bug:
https://kubernetes.slack.com/archives/C2P1JHS2E/p1575565526161900

we did fix a regression.
there is still the use case for pullPolicy = Never or allowing users skip prepull if we exposed it as a sub-phase of preflight.

[BenTheElder]

+1 @neolit123 xref kubernetes/kubernetes#90326

Elaborating a little ...

AFAIK kubeadm works in air gapped environment if the image exists locally; users also have options to influence image name (via image repository/image tag fields)

This is not sufficient, you cannot currently configure the pause image to match your CRI. In an airgapped environment this leads to: kubernetes/kubernetes#90326

#2020 would be one way to fix that, but alternatively in an airgapped environment I just don't want kubeadm trying to pull images at all, full stop.

If the cluster fails to come up due to missing images that should be relatively easy to diagnose. Pulling was never going to help. I know that I'm not going to pull in an airgapped env so I'd prefer to be able to tell kubeadm exactly that and skip this entirely.

I might not want to skip all preflight checks on a serious cluster though.

(btw though, there's another major potential issue in airgapped env currently that the already existing images can be evicted :/ I'm going to try to revive kubernetes/enhancements#1007)
EDIT: see kubernetes/enhancements#1717

[xlgao-zju]

@neolit123 shall add a flag (maybe pullPolicy ) to control the behavior of prepulls in pre-flight? os, users can skip the prepulls in flight, instead of just skipping the pre-flight.

and I am happy to help with this.

[rosti]

@xlgao-zju thanks for your help!

This is on hold now. No new command line flags should be added. The change needs to be part of a new kubeadm config version if done.
But, frankly, we haven't reached a decision if this actually needs to be done or not.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[neolit123]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[neolit123]

/remove-lifecycle stale

[neolit123]

@fabriziopandini and me agreed that this seems like an good change for 1.22 / v1beta3.

having a new field under NodeRegistrationOptions.imagePullPolicy that has values of Always, IfNotPresent, Never (same as k8s) makes sense:
https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#NodeRegistrationOptions
currently kubeadm does an implicit IfNotPresent.

[pacoxu]

/assign

[wangyysde]

/cc @wangyysde