Change or Add VM to worker pool - Failure after Master nodes change VM SKU #324

rmelilloii · 2024-02-29T12:32:39Z

rmelilloii
Feb 29, 2024

Hello and good morning/afternoon!

Opening a dedicated thread here as I believe it deserves dedicated attention. @vitobotta , I will delete my comment on the discussion 158. Any help or direction are very welcome.

Case: Hetzner Cloud decommissioned old VM SKU that was in use by our Master nodes. Each Master node had the VM SKU changed via Hetzner Cloud Console. After that, we can't add a new worker pool.

Details so far:

deployed cluster 3 years ago (2021)
- hetzner-k3s adds the VM SKU to the name of the node
- Hetzner VM for Master pool and one of the Worker pools was "rescaled".
  - the rescale was done from Hetzner console as per Hetzner instruction. also, afaik, hetzner-k3s does not support it, please correct me.
- now I need to deploy a new Worker pool and my deploy yaml will not accept the use of a SKU that does not exist in Hetzner and will look for the VM name to decide if it should deploy a new set of NMs for Master or not. To make possible to deploy the new worker pool, I then rename the VM to match the new SKU.
  - the new worker poll and its VMs are deployed, the existent Master nodes are not touched.
    - But it creates route problem on the nodes that had the name/sku changed:
      FailedToCreateRoute Node rome-test-cx21-master1 (combined from similar events): Could not create route 4c1fd58a-028d-4fc4-b706-ac9212a922ff 10.244.0.0/24 for node rome-test-cx21-master1 after 1.933111343s: hcloud/CreateRoute: hcops/AllServersCache.ByName: rome-test-cx21-master1 hcops/AllServersCache.getCache: not found
      To try to "fix it" I am trying to change the name on labels and annotations.
      My question is @vitobotta, is there anything to do on my case or the only solution is to deploy the new nodes manually?

I have the impression that IF the VM SKU is not added to the VM name during deploy, this would not be an issue.

additional info: I think this is a valid workaround for my case:

A: to be able to deploy I need to inform the "new" existent VM SKU on the deploy yaml for hetzner-k3s
B: to avoid that a new set of masters are deployed I need to change the VM name in hetzner
C: I can deploy the new VMs using hetzner-k3s, it will deploy only the new stuff as it will be able to find all the "pre-existent VMs"
D: I change the VM name back to fix the Kubernetes route issue

This for some reason gave me timeout on the SSH connectivity validation (worked on a test cluster)

With the timeout my next step was to comment out the other worker pools, leaving only "master" and the new worker pool. Then, I got the new VMs deployed, got the SSH validation, but it times out there.

rmelilloii · 2024-02-29T13:08:15Z

rmelilloii
Feb 29, 2024
Author

@vitobotta , probably related to this one?
#141

I can connect directly via SSH, and the auth.log is:

Last login: Thu Feb 29 12:59:07 2024 from 193.28.90.54
root@us-expireon-prod-cpx51-pool-pg-coc-large-worker1:~# tail -f /var/log/auth.log
Feb 29 13:00:44 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 systemd: pam_unix(systemd-user:session): session opened for user root(uid=0) by (uid=0)
Feb 29 13:00:44 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 sshd[941]: Received disconnect from 193.28.90.54 port 61005:11: bye
Feb 29 13:00:44 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 sshd[941]: Disconnected from user root 193.28.90.54 port 61005
Feb 29 13:00:44 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 sshd[941]: pam_unix(sshd:session): session closed for user root
Feb 29 13:00:44 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 systemd-logind[833]: Session 1 logged out. Waiting for processes to exit.
Feb 29 13:00:44 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 systemd-logind[833]: Removed session 1.
Feb 29 13:04:26 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 sshd[1030]: Accepted publickey for root from 193.28.90.54 port 49475 ssh2: RSA SHA256:a8ghVqQ+0C2XdvXq6jiOUdP54Iq3kbkNuZ63u/+JmEc
Feb 29 13:04:26 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 sshd[1030]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Feb 29 13:04:26 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 systemd-logind[833]: New session 3 of user root.
Feb 29 13:04:26 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 systemd: pam_unix(systemd-user:session): session opened for user root(uid=0) by (uid=0)
Feb 29 13:04:58 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 sshd[1104]: Accepted publickey for root from 193.28.90.54 port 60949 ssh2: RSA SHA256:a8ghVqQ+0C2XdvXq6jiOUdP54Iq3kbkNuZ63u/+JmEc
Feb 29 13:04:58 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 sshd[1104]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Feb 29 13:04:58 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 systemd-logind[833]: New session 5 of user root.
Feb 29 13:04:59 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 sshd[1104]: Received disconnect from 193.28.90.54 port 60949:11: bye
Feb 29 13:04:59 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 sshd[1104]: Disconnected from user root 193.28.90.54 port 60949
Feb 29 13:04:59 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 sshd[1104]: pam_unix(sshd:session): session closed for user root
Feb 29 13:04:59 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 systemd-logind[833]: Session 5 logged out. Waiting for processes to exit.
Feb 29 13:04:59 us-expireon-prod-cpx51-pool-pg-coc-large-worker1 systemd-logind[833]: Removed session 5.

with ssh_agent = enabled and port = 22:
we have a loop of connections/disconnections.

Feb 29 16:00:36 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3000]: Disconnected from 193.28.90.54 port 60924 [preauth]
Feb 29 16:00:38 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3002]: Received disconnect from 193.28.90.54 port 58533:11: bye [preauth]
Feb 29 16:00:38 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3002]: Disconnected from 193.28.90.54 port 58533 [preauth]
Feb 29 16:00:39 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3004]: Received disconnect from 193.28.90.54 port 59006:11: bye [preauth]
Feb 29 16:00:39 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3004]: Disconnected from 193.28.90.54 port 59006 [preauth]
Feb 29 16:00:40 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3008]: Received disconnect from 193.28.90.54 port 60950:11: bye [preauth]
Feb 29 16:00:40 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3008]: Disconnected from 193.28.90.54 port 60950 [preauth]
Feb 29 16:00:41 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3006]: Invalid user shing from 128.199.243.189 port 35246
Feb 29 16:00:41 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3006]: Received disconnect from 128.199.243.189 port 35246:11: Bye Bye [preauth]
Feb 29 16:00:41 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3006]: Disconnected from invalid user shing 128.199.243.189 port 35246 [preauth]
Feb 29 16:00:42 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3010]: Received disconnect from 193.28.90.54 port 59036:11: bye [preauth]
Feb 29 16:00:42 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3010]: Disconnected from 193.28.90.54 port 59036 [preauth]
Feb 29 16:00:44 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3012]: Received disconnect from 193.28.90.54 port 60922:11: bye [preauth]
Feb 29 16:00:44 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3012]: Disconnected from 193.28.90.54 port 60922 [preauth]
Feb 29 16:00:47 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3014]: Received disconnect from 193.28.90.54 port 58445:11: bye [preauth]
Feb 29 16:00:47 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3014]: Disconnected from 193.28.90.54 port 58445 [preauth]
Feb 29 16:00:48 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3016]: Received disconnect from 193.28.90.54 port 58964:11: bye [preauth]
Feb 29 16:00:48 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3016]: Disconnected from 193.28.90.54 port 58964 [preauth]
Feb 29 16:00:53 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3018]: Received disconnect from 193.28.90.54 port 60954:11: bye [preauth]
Feb 29 16:00:53 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3018]: Disconnected from 193.28.90.54 port 60954 [preauth]
Feb 29 16:00:53 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3019]: Received disconnect from 193.28.90.54 port 58990:11: bye [preauth]
Feb 29 16:00:53 us-expireon-prod-ccx63-pool-pg-coc-large-worker1 sshd[3019]: Disconnected from 193.28.90.54 port 58990 [preauth]

my config: (again, this is an old existent cluster), 9 months ago we were able to add a new worker pool, but it was before the VM SKU change.

---
hetzner_token: ***
cluster_name: ***
kubeconfig_path: "./kubeconfig"
k3s_version: v1.25.9+k3s1
#public_ssh_key_path: "~/.ssh/expUsProd/id_rsa.pub"
#private_ssh_key_path: "~/.ssh/expUsProd/id_rsa"
public_ssh_key_path: "~/.ssh/id_rsa_exp-us-prod.pub"
private_ssh_key_path: "~/.ssh/id_rsa_exp-us-prod"
#use_ssh_agent: false
#ssh_port: 22
ssh_allowed_networks:
  - 0.0.0.0/0
api_allowed_networks:
  - 0.0.0.0/0
schedule_workloads_on_masters: false
private_network_subnet: 10.0.0.0/16
disable_flannel: false # set to true if you want to install a different CNI
cloud_controller_manager_manifest_url: "https://raw.githubusercontent.com/hetznercloud/hcloud-cloud-controller-manager/v1.18.0/deploy/ccm-networks.yaml"
csi_driver_manifest_url: "https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.5.1/deploy/kubernetes/hcloud-csi.yml"
system_upgrade_controller_manifest_url: "https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml"
masters_pool:
  instance_type: ccx23
  instance_count: 3
  location: ash
  image: ubuntu-22.04
worker_node_pools:
#- name: largeworker
#  instance_type: ccx53
#  instance_count: 3
#  location: ash
#  image: ubuntu-22.04
#- name: postgres-nodes
#  instance_type: cpx51
#  instance_count: 8
#  location: ash
#  image: ubuntu-22.04
#- name: postgres-nodes-2
#  instance_type: cpx51
#  instance_count: 8
#  location: ash
#  image: ubuntu-22.04
#- name: postgres-coc-nodes1
#  instance_type: cpx51
#  instance_count: 2
#  location: ash
#  image: ubuntu-22.04
#- name: elastic-nodes
#  instance_type: cpx51
#  instance_count: 3
#  location: ash
#  image: ubuntu-22.04
- name: pg-coc-large
  instance_type: ccx63
  instance_count: 4
  location: ash
  image: ubuntu-22.04
  labels:
    - key: pg-coc-large
      value: enabled
post_create_commands:
- apt update
- apt upgrade -y
- apt autoremove -y
- shutdown -r now
enable_encryption: true

7 replies

vitobotta Apr 12, 2024
Maintainer

Oh so you just changed the names of the servers in the Hetzner console and the instance type in the config file?

rmelilloii Apr 12, 2024
Author

yep, I change it in hetzner (VMs), It will prompt an error on kubernetes > I run the CLI also, informing the new/current VM SKU.
It works fine for small clusters. (no need for manual k3s install and join).
It times out for bigger clusters. (here I need to install k3s and join).

Both scenarios, do not present any issue on the Kubernetes cluster after deploy.

vitobotta Apr 12, 2024
Maintainer

Good to know! I will add a note about this as a solution for existing clusters when I make the new release. Thanks!

rmelilloii Apr 12, 2024
Author

Thank you, really.
To give you a more approachable info, this is what I wrote on my internal wiki:

Workaround issue 1
to add a new set of VMs
we need to rename the VM name via Hetzner Console to match the new VM SKU. (this will create issues, so the change must be temporary)
the CLI tool uses the name to search for existent resources and will create a new set in case it doesn't find them.
we change the VM SKU on the CLI tool yaml for the affected ones and then, to be safe, we comment all the other existent VMs, leaving only Master and the new ones.
now, after we run the CLI tool, the new VMs will be created, but the k3s service will not. It must be done manually. We also need to check/compare all Tags in Kubernetes and in Hetzner Cloud to make sure they were added.
after the deploy we change the Names of the VMs back to their original name in Hetzner Console.
we access all VMs via SSH and make sure they are up to date. Now we can proceed and add them to the kubernetes cluster as agents/workers or masters.

As we are here, in case it is relevant for you too, also part or our internal wiki

Deployment issue 2
Hetzner has some Public IPv4s that are blacklisted and there is nothing we can do. So far we only had one issue, when accessing "docker.elastic.co". It would then fail to deploy Elasticsearch.

How to test: SSH into the VM and run: curl -v docker.elastic.co, in case it times out, we have a blocked IP.

Workaround issue 2
The official workaround from Hetzner is:
"If this leads to issues for you, please create a Snapshot of the server with the incorrect IP location.
Then create a new server with this Snapshot. You can then delete the server with the incorrect IP location.
I am afraid we can not offer a different solution."

This workaround is rubbish 😃 We found out that we can simply unassign the IP and assign a new one (might need to try a couple of times until we get a new one without the issue). For that, we access the Hetzner console, go to the VM that has the IP with the issue. The VM needs to be switched off. With that we can unassign the IP, add a new one and we test.

vitobotta Apr 12, 2024
Maintainer

On workaround for issue 2, if you remove the ip and assign a new one does it affect the functionality of the node to the cluster or not at all? Does it require rebooting the node?

vitobotta · 2024-04-12T13:46:23Z

vitobotta
Apr 12, 2024
Maintainer

I am planning to fix the SKU issue by removing it from the instance names for new clusters. Unfortunately for existing clusters there is not much that can be done. I mean for worker nodes you can just add new pools with the new instance type, migrate stuff and delete the old node pools if you want, but for masters it's not so easy.

0 replies

rmelilloii · 2024-04-12T18:22:57Z

rmelilloii
Apr 12, 2024
Author

It does not affect the cluster, on my understanding as it uses mainly the private IP. Yes, it is a requirement to shutdown the VM. That is why I suggest doing it at deploy :) Romulo II Melillo *Mobile:* +44 07367437841 *Skype:* rmelilloii Linkedin <https://uk.linkedin.com/in/rmelilloii>

…

On Fri, 12 Apr 2024 at 17:14, Vito Botta ***@***.***> wrote: On workaround for issue 2, if you remove the ip and assign a new one does it affect the functionality of the node to the cluster or not at all? Does it require rebooting the node? — Reply to this email directly, view it on GitHub <#324 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG3HX5D2XVXMQ26XBID5D73Y5AB6XAVCNFSM6AAAAABD73DP5WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TAOJYGA2TK> . You are receiving this because you authored the thread.Message ID: ***@***.***>

1 reply

vitobotta Apr 12, 2024
Maintainer

Gotcha. Thanks for the info, very useful :)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change or Add VM to worker pool - Failure after Master nodes change VM SKU #324

{{title}}

Replies: 3 comments 8 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Change or Add VM to worker pool - Failure after Master nodes change VM SKU #324

rmelilloii Feb 29, 2024

Replies: 3 comments · 8 replies

rmelilloii Feb 29, 2024 Author

vitobotta Apr 12, 2024 Maintainer

rmelilloii Apr 12, 2024 Author

vitobotta Apr 12, 2024 Maintainer

rmelilloii Apr 12, 2024 Author

vitobotta Apr 12, 2024 Maintainer

vitobotta Apr 12, 2024 Maintainer

rmelilloii Apr 12, 2024 Author

vitobotta Apr 12, 2024 Maintainer

rmelilloii
Feb 29, 2024

Replies: 3 comments 8 replies

rmelilloii
Feb 29, 2024
Author

vitobotta Apr 12, 2024
Maintainer

rmelilloii Apr 12, 2024
Author

vitobotta Apr 12, 2024
Maintainer

rmelilloii Apr 12, 2024
Author

vitobotta Apr 12, 2024
Maintainer

vitobotta
Apr 12, 2024
Maintainer

rmelilloii
Apr 12, 2024
Author

vitobotta Apr 12, 2024
Maintainer