Kubespray GCE PVC provisioning is not working on k8s 1.22 in release 2.18

[lmercl]

Hello,
looks like on k8s 1.22 in kubespray release 2.18 is broken GCE PVC provisioning and usage.

The issue is that CSI controller looks to v1beta1 k8s API in used GKE images. I also tried to use the newest GKE images (version below) but issue is the same. When I changed images to kubernetes-sig images (version below), issue with API was resolved and provisioning of PVs is working, but there is another issue, that CSI mount PVs with permission 755 so it means pods which are not running on root account are not able to write to these PVs by default.

I0218 15:43:10.984568       1 reflector.go:188] Listing and watching *v1beta1.CSINode from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108
E0218 15:43:10.987491       1 reflector.go:156] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1beta1.CSINode: the server could not find the requested resource
I0218 15:43:11.987687       1 reflector.go:188] Listing and watching *v1beta1.CSINode from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108
E0218 15:43:11.991571       1 reflector.go:156] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1beta1.CSINode: the server could not find the requested resource

ORIGINAL kubespray's images:
gcp_pd_csi_image_repo: "gke.gcr.io"
gcp_pd_csi_driver_image_tag: "v0.7.0-gke.0"
gcp_pd_csi_provisioner_image_tag: "v1.5.0-gke.0"
gcp_pd_csi_attacher_image_tag: "v2.1.1-gke.0"
gcp_pd_csi_resizer_image_tag: "v0.4.0-gke.0"
gcp_pd_csi_registrar_image_tag: "v1.2.0-gke.0"

The newest images on gke.gcr.io:
gcp_pd_csi_image_repo: "gke.gcr.io"
gcp_pd_csi_driver_image_tag: "v1.4.0-gke.0"
gcp_pd_csi_provisioner_image_tag: "v1.6.0-gke.0"
gcp_pd_csi_attacher_image_tag: "v2.2.0-gke.0"
gcp_pd_csi_resizer_image_tag: "v0.5.0-gke.0"
gcp_pd_csi_registrar_image_tag: "v1.3.0-gke.0"

I also tried another images from kubernetes-sig:
k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
k8s.gcr.io/sig-storage/csi-resizer:v1.3.0
k8s.gcr.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0
k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0

Environment:

• Cloud provider or hardware configuration: GCE

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

• Version of Ansible (ansible --version):

• Version of Python (python --version):

Kubespray version (commit) (git rev-parse --short HEAD): 2.18

Network plugin used: N/a

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

Command used to invoke ansible:

Output of ansible run:

Anything else do we need to know: