seldon-core-analytics helm installation fails

[spi-x-i]

Describe the bug

Running install procedure for seldon-core-analytics fails; namely, it fails on the execution of helm install. This is probably due to unsupported APIs rbac.authorization.k8s.io/v1beta1 by Kubernetes 1.22.x for objects ClusterRole, ClusterRoleBinding, Role, RoleBinding.

To reproduce

1. Install k8s 1.22 with minikube
2. Install seldon 1.13.x
3. helm install of seldon-core-analytics by following the docs.

$ kubectl create namespace seldon-system

$ helm install seldon-core-analytics seldonio/seldon-core-analytics --namespace seldon-system

The error:

Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "ClusterRole" in version "rbac.authorization.k8s.io/v1beta1", unable to recognize "": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1", unable to recognize "": no matches for kind "Role" in version "rbac.authorization.k8s.io/v1beta1", unable to recognize "": no matches for kind "RoleBinding" in version "rbac.authorization.k8s.io/v1beta1"]

Expected behaviour

It should install seldon-core-analytics

Environment

• Cluster provider: minikube
• Kubernetes cluster version

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}

• Deployed Seldon System Images:

value: docker.io/seldonio/seldon-core-executor:1.13.1
image: docker.io/seldonio/seldon-core-operator:1.13.1

[spi-x-i]

Probably, the easiest workaround is to downgrade Kubernetes to 1.21.x.

Any chance you guys plan to tackle this? Thank you

[agrski]

Hi @spi-x-i, I've been able to reproduce your issue by running the following:

$ kind create cluster --name analytics-1.22-1.13 --image kindest/node:v1.22.7
$ helm install seldon-core-analytics seldonio/seldon-core-analytics --version v1.13.1

As RBAC v1 has been available since the long-deprecated Kubernetes 1.8, it seems reasonable to bump this to use the v1 (rather than v1beta1) APIs.
I'll look into this shortly.

[agrski]

Looks like this PR already made that change back in October, so I wonder why this isn't working

[agrski]

Looks like it's coming from the underlying Prometheus chart at version 11.4.0, which references v1beta1.

To solve this, we'd need to bump the dependency version, which is a little bit riskier than just changing RBAC API versions. I'll continue to look into this.

[axsaucedo]

@spi-x-i thank you for the heads up - we're looking to address this in this next version of Seldon Core by moving into using the Prometheus Operator instead via #3886

[agrski]

Bumping the Prometheus chart to 15.5.x was successful:

$ git diff Chart.yaml
diff --git a/helm-charts/seldon-core-analytics/Chart.yaml b/helm-charts/seldon-core-analytics/Chart.yaml
index 40a525e9a..51158abd5 100644
--- a/helm-charts/seldon-core-analytics/Chart.yaml
+++ b/helm-charts/seldon-core-analytics/Chart.yaml
@@ -7,8 +7,8 @@ dependencies:
   version: ~5.1.4
 - condition: prometheus.enabled
   name: prometheus
-  repository: https://charts.helm.sh/stable
-  version: ~11.4.0
+  repository: https://prometheus-community.github.io/helm-charts
+  version: ~15.5.0

$ helm dependency update

$ helm upgrade --install seldon-core-analytics ./ --set grafana.enabled=false

Note that the Grafana dependency (optional) still has some reference(s) to RBAC v1beta1, hence the need to disable it here (or bump the version).

@spi-x-i As a workaround, you can apply the same locally by checking out seldon-core and applying the above changes.

[spi-x-i]

Thank you @agrski @axsaucedo for your quick response. I'll go with the workaround while we wait for the next release.

[axsaucedo]

Closing as workaround provided and proper resolution via #3886