Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

etcd datastore migration doesn't succeed #375

Closed
Heiko-san opened this issue Sep 22, 2023 · 7 comments
Closed

etcd datastore migration doesn't succeed #375

Heiko-san opened this issue Sep 22, 2023 · 7 comments
Labels
invalid This doesn't seem right

Comments

@Heiko-san
Copy link

Hi

We started a migration of a newly created cluster from one newly ceated kamaji-etcd datastore to another one.
(We did this to test the fix for issue #350, which seems to be not an issue anymore.)

The migration pod is created and completes and the TCP gets patched into "Migrating":

k -n kamaji-system get pod
NAME                                              READY   STATUS      RESTARTS   AGE
capi-kamaji-controller-manager-7bbd5c8c95-sz2gs   1/1     Running     0          41h
kamaji-64c49b885b-8d72p                           1/1     Running     0          41h
migrate-kpoc-tenant3-kpoc-tenant3-r26sl           0/1     Completed   0          15m

kg tcp                    
NAME           VERSION   STATUS      CONTROL-PLANE ENDPOINT   KUBECONFIG                      DATASTORE           AGE
kpoc-tenant3   v1.25.7   Migrating   10.123.1.41:6443         kpoc-tenant3-admin-kubeconfig   kpoc-tenant3-etcd   26m

Logs of the Job:

2023/09/22 08:51:17 maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined
2023-09-22T08:51:17Z	INFO	generating the controller-runtime client
2023-09-22T08:51:18Z	INFO	retrieving the TenantControlPlane
2023-09-22T08:51:18Z	INFO	retrieving the TenantControlPlane used DataStore
2023-09-22T08:51:18Z	INFO	retrieving the target DataStore
2023-09-22T08:51:18Z	INFO	generating the origin storage connection
2023-09-22T08:51:18Z	INFO	generating the target storage connection
2023-09-22T08:51:18Z	INFO	migration from origin to target started
2023-09-22T08:51:33Z	INFO	migration completed

The problem is: The migration never comes to an end, although the job completed.
And if you have a look at the pods of the TCP, they still point to the old datastore.

Logs of kamaji:

2023-09-22T08:51:16Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "0ea47be8-bd0d-403c-baa5-a3ec25b2d656", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:16Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "0ea47be8-bd0d-403c-baa5-a3ec25b2d656", "allowed": true}
2023-09-22T08:51:16Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "235b1b26-7b2c-453d-bbb8-b64ab4b496d1", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:16Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "235b1b26-7b2c-453d-bbb8-b64ab4b496d1", "allowed": true}
2023-09-22T08:51:16Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseBootstrapToken	start processing
2023-09-22T08:51:16Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseBootstrapToken	reconciliation completed
2023-09-22T08:51:16Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.konnectivity_agent	start processing	{"resource": "konnectivity-agent"}
2023-09-22T08:51:16Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.kube_proxy	start processing
2023-09-22T08:51:16Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.coredns	start processing
2023-09-22T08:51:16Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseUploadConfigKubeadm	start processing
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
2023-09-22T08:51:16Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseUploadConfigKubelet	start processing
2023-09-22T08:51:16Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "6ce5c806-de63-4c5d-b478-115635d247dd", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:16Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "6ce5c806-de63-4c5d-b478-115635d247dd", "allowed": true}
2023-09-22T08:51:17Z	INFO	migrate has been configured	{"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kpoc-tenant3","namespace":"kpoc-tenant3"}, "namespace": "kpoc-tenant3", "name": "kpoc-tenant3", "reconcileID": "c2a672c5-9aec-403f-961d-993dd97c6349"}
2023-09-22T08:51:17Z	INFO	requested enqueuing back	{"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kpoc-tenant3","namespace":"kpoc-tenant3"}, "namespace": "kpoc-tenant3", "name": "kpoc-tenant3", "reconcileID": "c2a672c5-9aec-403f-961d-993dd97c6349", "resources": "migrate"}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseBootstrapToken	start processing
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseBootstrapToken	reconciliation completed
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "ba0a9aaf-de2c-487c-8991-58a9b62dd879", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "ba0a9aaf-de2c-487c-8991-58a9b62dd879", "allowed": true}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.konnectivity_agent	resource processed	{"resource": "konnectivity-agent"}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.konnectivity_agent	start processing	{"resource": "konnectivity-sa"}
2023-09-22T08:51:17Z	DEBUG	sentinel error, enqueuing back request	{"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kpoc-tenant3","namespace":"kpoc-tenant3"}, "namespace": "kpoc-tenant3", "name": "kpoc-tenant3", "reconcileID": "dcb293c1-d732-4a1a-9835-5cbbc548cc0b", "error": "cannot continue reconciliation, the current TenantControlPlane is still in migration status"}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "b08d58b6-6650-4d9d-9d52-dc31924748d0", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "b08d58b6-6650-4d9d-9d52-dc31924748d0", "allowed": true}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "f9681565-41c8-43d6-b2ff-510f71c04b62", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "f9681565-41c8-43d6-b2ff-510f71c04b62", "allowed": true}
2023-09-22T08:51:17Z	DEBUG	sentinel error, enqueuing back request	{"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kpoc-tenant3","namespace":"kpoc-tenant3"}, "namespace": "kpoc-tenant3", "name": "kpoc-tenant3", "reconcileID": "031d3116-3a2e-40b8-8edd-b770ad1733ea", "error": "cannot continue reconciliation, the current TenantControlPlane is still in migration status"}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "77e5808c-b7c1-4bce-abe6-c9503af1b62f", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "77e5808c-b7c1-4bce-abe6-c9503af1b62f", "allowed": true}
2023-09-22T08:51:17Z	DEBUG	sentinel error, enqueuing back request	{"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kpoc-tenant3","namespace":"kpoc-tenant3"}, "namespace": "kpoc-tenant3", "name": "kpoc-tenant3", "reconcileID": "6b2e09a5-e476-4eaa-9e94-4f75f3305ef2", "error": "cannot continue reconciliation, the current TenantControlPlane is still in migration status"}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseBootstrapToken	start processing
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseBootstrapToken	reconciliation completed
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "7c7457aa-2cf1-48d7-927b-58eb421aebe7", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "7c7457aa-2cf1-48d7-927b-58eb421aebe7", "allowed": true}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "8077bd37-913e-454a-bb0c-d641f26be4bc", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "8077bd37-913e-454a-bb0c-d641f26be4bc", "allowed": true}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "c25d42a0-33dc-4f0a-abc2-cc7bb3008418", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "c25d42a0-33dc-4f0a-abc2-cc7bb3008418", "allowed": true}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.konnectivity_agent	start processing	{"resource": "konnectivity-clusterrolebinding"}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseUploadConfigKubeadm	reconciliation completed
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseUploadConfigKubeadm	start processing
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseUploadConfigKubelet	reconciliation completed
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseUploadConfigKubelet	start processing
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "d8e8c645-7fd9-43a5-89d0-f577fa85254a", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "d8e8c645-7fd9-43a5-89d0-f577fa85254a", "allowed": true}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "cc4f956d-0ca3-4902-ab78-9913c0dd5bfe", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "cc4f956d-0ca3-4902-ab78-9913c0dd5bfe", "allowed": true}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "385eefc5-ab14-4377-8be3-1ea45adcb463", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "385eefc5-ab14-4377-8be3-1ea45adcb463", "allowed": true}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.konnectivity_agent	reconciliation completed
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.konnectivity_agent	start processing	{"resource": "konnectivity-agent"}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "d7046f54-9756-40a4-a3f3-1863eb9f36f1", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "d7046f54-9756-40a4-a3f3-1863eb9f36f1", "allowed": true}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.coredns	reconciliation completed
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.coredns	start processing
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.kube_proxy	reconciliation completed
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.kube_proxy	start processing
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "3f4ec054-aa01-486a-bc59-745938918b89", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.konnectivity_agent	resource processed	{"resource": "konnectivity-agent"}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.konnectivity_agent	start processing	{"resource": "konnectivity-sa"}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "3f4ec054-aa01-486a-bc59-745938918b89", "allowed": true}
2023-09-22T08:51:17Z	INFO	kpoc-tenant3 has been reconciled	{"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kpoc-tenant3","namespace":"kpoc-tenant3"}, "namespace": "kpoc-tenant3", "name": "kpoc-tenant3", "reconcileID": "cbea8654-dd51-40eb-87da-f01fb9c892b5"}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseUploadConfigKubelet	reconciliation completed
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "9bee8855-3233-46af-a4b9-5b99504aa550", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "9bee8855-3233-46af-a4b9-5b99504aa550", "allowed": true}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "86220444-cf82-40a2-9a78-638eff52e862", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "86220444-cf82-40a2-9a78-638eff52e862", "allowed": true}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "ceecc274-45ac-4c4d-9a62-6c581a993825", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "ceecc274-45ac-4c4d-9a62-6c581a993825", "allowed": true}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.PhaseUploadConfigKubeadm	reconciliation completed
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "f40ddffe-6a37-470f-8448-312ea81caa95", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "f40ddffe-6a37-470f-8448-312ea81caa95", "allowed": true}
2023-09-22T08:51:17Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.konnectivity_agent	start processing	{"resource": "konnectivity-clusterrolebinding"}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "8aaf4dff-2d8e-4525-973b-d92142363a26", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "8aaf4dff-2d8e-4525-973b-d92142363a26", "allowed": true}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "7ea1857c-6bab-41df-a3f3-a1fb16e79980", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:17Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "7ea1857c-6bab-41df-a3f3-a1fb16e79980", "allowed": true}
2023-09-22T08:51:18Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.konnectivity_agent	reconciliation completed
2023-09-22T08:51:18Z	INFO	kpoc-tenant3 has been reconciled	{"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kpoc-tenant3","namespace":"kpoc-tenant3"}, "namespace": "kpoc-tenant3", "name": "kpoc-tenant3", "reconcileID": "56945138-6edd-4a5e-9561-8b613fd21396"}
2023-09-22T08:51:18Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.kube_proxy	reconciliation completed
2023-09-22T08:51:18Z	INFO	soot_kpoc-tenant3_kpoc-tenant3.coredns	reconciliation completed
2023-09-22T08:51:18Z	INFO	kpoc-tenant3 has been reconciled	{"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kpoc-tenant3","namespace":"kpoc-tenant3"}, "namespace": "kpoc-tenant3", "name": "kpoc-tenant3", "reconcileID": "60e22df4-cb82-45f4-8595-7e69fd2f6632"}
2023-09-22T08:51:18Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "81d13929-20d3-4226-9501-4330a90b8fd3", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:18Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "81d13929-20d3-4226-9501-4330a90b8fd3", "allowed": true}
2023-09-22T08:51:18Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "7aa12854-8171-4342-ab4b-1915a1a22095", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:18Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "7aa12854-8171-4342-ab4b-1915a1a22095", "allowed": true}
2023-09-22T08:51:19Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "8523ffa8-c238-4de1-9095-cbb9d3692fe8", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:19Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "8523ffa8-c238-4de1-9095-cbb9d3692fe8", "allowed": true}
2023-09-22T08:51:19Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "7c108b1d-21a3-4086-9edd-77c7830db38f", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:19Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "7c108b1d-21a3-4086-9edd-77c7830db38f", "allowed": true}
2023-09-22T08:51:22Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "3335eb64-d13a-4dcb-91a2-5ec403d420ac", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:22Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "3335eb64-d13a-4dcb-91a2-5ec403d420ac", "allowed": true}
2023-09-22T08:51:22Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "fc640036-5e56-414f-ad04-8a60a8703afc", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:22Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "fc640036-5e56-414f-ad04-8a60a8703afc", "allowed": true}
2023-09-22T08:51:27Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "d8762b6b-d488-40dd-a68e-8ecc7d760606", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:27Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "d8762b6b-d488-40dd-a68e-8ecc7d760606", "allowed": true}
2023-09-22T08:51:27Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "70481d71-86d4-4bea-992a-149514884c2c", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:27Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "70481d71-86d4-4bea-992a-149514884c2c", "allowed": true}
2023-09-22T08:51:38Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "b4abb2c8-cba0-49e7-800f-eef71aa78e8d", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:38Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "b4abb2c8-cba0-49e7-800f-eef71aa78e8d", "allowed": true}
2023-09-22T08:51:38Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "14dad903-a388-4ddc-806b-6ba3b4f46f63", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:38Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "14dad903-a388-4ddc-806b-6ba3b4f46f63", "allowed": true}
2023-09-22T08:51:58Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "0b3819d3-0838-45ca-af06-b54518801b6e", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:58Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "0b3819d3-0838-45ca-af06-b54518801b6e", "allowed": true}
2023-09-22T08:51:58Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "UID": "01c3ee9b-37bb-413e-a400-cc31f28098e1", "kind": "kamaji.clastix.io/v1alpha1, Kind=TenantControlPlane", "resource": {"group":"kamaji.clastix.io","version":"v1alpha1","resource":"tenantcontrolplanes"}}
2023-09-22T08:51:58Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-kamaji-clastix-io-v1alpha1-tenantcontrolplane", "code": 200, "reason": "update operation allowed", "UID": "01c3ee9b-37bb-413e-a400-cc31f28098e1", "allowed": true}

Versions:

helm list 

NAME                	NAMESPACE   	REVISION	UPDATED                                 	STATUS  	CHART            	APP VERSION
kpoc-tenant3-etcd   	kpoc-tenant3	1       	2023-09-22 10:39:17.845855673 +0200 CEST	deployed	kamaji-etcd-0.4.0	3.5.6      
kpoc-tenant3new-etcd	kpoc-tenant3	1       	2023-09-22 10:40:55.212783699 +0200 CEST	deployed	kamaji-etcd-0.4.0	3.5.6

helm list

NAME  	NAMESPACE    	REVISION	UPDATED                                 	STATUS  	CHART        	APP VERSION
kamaji	kamaji-system	1       	2023-09-20 17:53:21.502222386 +0200 CEST	deployed	kamaji-0.12.5	v0.3.4

clusterctl upgrade plan                   ✔ 
Checking new release availability...

Latest release available for the v1beta1 API Version of Cluster API (contract):

NAME                     NAMESPACE                       TYPE                     CURRENT VERSION   NEXT VERSION
bootstrap-kubeadm        capi-kubeadm-bootstrap-system   BootstrapProvider        v1.5.1            Already up to date
control-plane-kamaji     kamaji-system                   ControlPlaneProvider     v0.3.0            v0.3.1
cluster-api              capi-system                     CoreProvider             v1.5.1            Already up to date
infrastructure-vsphere   capv-system                     InfrastructureProvider   v1.8.1            Already up to date
@Heiko-san Heiko-san mentioned this issue Sep 22, 2023
Closed
@prometherion
Copy link
Member

👋🏻 may I ask you how do you change the DataStore reference?

Since Cluster API is mentioned here, I take for granted you're doing that by editing the KamajiControlPlane resource, isn't it?

@Heiko-san
Copy link
Author 

kubectl patch --type merge tcp kpoc-tenant3 -p '{"spec": {"dataStore": "kpoc-tenant3new-etcd"}}'

@prometherion
Copy link
Member

As I suggested in another comment, when TenantControlPlane objects are orchestrated through Cluster API, the KamajiControlPlane resource must be used, otherwise this would lead to race conditions like the following you're facing.

May I ask you to try to replicate the unsuccessful migration this time by editing the KamajiControlPlane resource, rather?

@Heiko-san
Copy link
Author

ah ok, yes , sorry I forgot that

@prometherion
Copy link
Member

No problem at all, mate! 🤗

Give it a try, and if it works, please, feel free to close the issue!

@Heiko-san
Copy link
Author

nevermind, it works as soon as you do it the right way, I'll close this

@prometherion prometherion added the invalid This doesn't seem right label Sep 22, 2023
@prometherion
Copy link
Member

I was thinking of a safe trigger to prevent direct changes to the TenantControlPlane resources managed by CAPI but I have no clue since if we block any changes, even CAPI is not able to patch the resource, as well as the Service Accounts running in the kube-system Namespace.

What we need, maybe, is just to make it more clear that once a TenantControlPlane object is created via Cluster API, the KamajiControlPlane resource must be the unique source of truth. 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
invalid This doesn't seem right
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@prometherion @Heiko-san