Too many machineDeployment generations

[prashanth26]

Issue

The machine deployment generations are constantly increasing with time. This causes issues with Gardener reconciliation.

Solution

Need to fix too many machine deployment updates

cc: @DockToFuture @adracus

[prashanth26]

These changes affect on shoots running as seeds negatively for seeds running Kubernetes version 1.13.x +.

Kindly refrain from migrating to seeds to 1.13.x until this issue is resolved.

The bug could be traced back to - kubernetes/kubernetes#69059

[adracus]

cc @rfranzke

[prashanth26]

• From my intial set of tests, the migration from /status field to /status subresource looks like it's not a direct migration.

• And if it's not a direct migration, we might do this migration along with the out of tree migration

• However, we still need to find a solution for seeds moving to seed 1.13 well before that if the above is true.

[rfranzke]

From my intial set of tests, the migration from /status field to /status subresource looks like it's not a direct migration.

Can you elaborate on this?

[prashanth26]

When a machine-deployment was created with the old status (with status as a field), and when an MCM which understands new status (subresource), I can see that updateStatus() seems to return an error saying machine/status (sub)resource not found.

My guess probably is because etcd has initailly stored the entire machine object as a single resource, and now suddenly it is split into two resources and the etcd doesn't know about the new status (sub)resource attached to the machine. I still have to verify this by looking to the Kubernetes docs to see why this change is not backward compatible.

I tried this experiment by migrating from an old MCM to new MCM on a Gardener setup and MCM started throwing these errors.

[rfranzke]

My guess probably is because etcd has initailly stored the entire machine object as a single resource, and now suddenly it is split into two resources and the etcd doesn't know about the new status (sub)resource attached to the machine.

No, /status subresource is only another REST API path. The object will not be split into two.

When a machine-deployment was created with the old status (with status as a field), and when an MCM which understands new status (subresource), I can see that updateStatus() seems to return an error saying machine/status (sub)resource not found.

Did you update the CRD and defining that you want to have the status subresource before starting new MCM?

[prashanth26]

Did you update the CRD and defining that you want to have the status subresource before starting new MCM?

No, I wasn't aware of it. Probably that is the reason then. Let me quickly try that out.

[rfranzke]

https://github.com/gardener/gardener/blob/master/charts/seed-bootstrap/templates/extensions/crd-operatingsystemconfig.yaml#L31-L32

[prashanth26]

This looks like it's working. Let me double check once more to be sure.

Thanks for the inputs @rfranzke 👍