Upgrade Kubernetes on cluster

[batpad]

Breaking out of #1885

We are currently running k8s version 1.25 . This will no longer be supported on Azure after the end of this year.

Version 1.27 is the latest version designated for Long Term Support and we should upgrade to that.

Steps:

• Look at CHANGELOGs for 1.26 and 1.27 and try and identify any changes we need to make to our templates due to backwards incompatible changes
• If needed, line up these changes in a PR
• Upgrade Staging cluster to 1.26
• Test all the things, including jobs, celery, etc.
• Repeat above steps to move staging to 1.27
• Do broader steps to test things on staging
• Upgrade production (@sunu for prod, should we also do it incrementally or should we go straight from 1.25 -> 1.27 if we've tested and made changes for 1.27 on staging?)

@sunu please add any notes or considerations we should keep in mind. cc @ranchodeluxe for visibility.

@szabozoltan69 @sunu can we plan to do this on Monday?

[batpad]

+cc @geohacker - I know you've done this on Azure before, so would be great to get your support / eyes on the exact steps we need to do in the console to perform the upgrade, etc.

[batpad]

This is done! Both prod and staging are now on the latest version in the 1.28.x series.

Thanks much @szabozoltan69 and @sunu !