Helm chart v2.0.0-rc2 fails to install on fresh k8s 1.18.8

[rhummelmose]

Attempting to install the chart fails with error

➜  duah-infrastructure git:(master) ✗ helm install keda kedacore/keda --version 2.0.0-rc2 --namespace keda       
Error: rendered manifests contain a resource that already exists. Unable to continue with install: CustomResourceDefinition "scaledjobs.keda.sh" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "keda"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "keda"

Expected Behavior

It is able to install on a fresh cluster

Actual Behavior

It does not install

Steps to Reproduce the Problem

1. Attempt to install

Specifications

• KEDA Version: 2.0.0-rc2
• Platform & Version: Azure, AKS, fresh as of 26/10/2020
• Kubernetes Version: 1.18.8

[tomkerkhove]

I'll take a look at this and have created #87 to avoid this in the future

[RemiGaudin]

Interestingly I have the same issue with version 2.0.0-rc2, but it's working fine with version 2.0.0-rc

[tomkerkhove]

@ahmelsayed do you have bandwidth to help here?

[jeffhollan]

Discussed in standup - hoping to get rid of the problem. @tomkerkhove shipped 2.0.0-rc3 works now, but it has helm 2. Need to fix so 2.0.0 can use helm 3 and not support helm 2.

@TsuyoshiUshio / @ahmelsayed let me know if you can help look at this. I may be able to spend some cycles tonight as well

[arschles]

I can give a hand here as well, will check it out later today

[tomkerkhove]

Thanks @arschles!

[rhummelmose]

Awesome guys. @arschles let me know if you can't find the time and I'll try to fix it during the weekend.

[jeffhollan]

Can confirm that 2.0.0-rc3 works, but it isn't published to the index so had to clone. Let me clean it up a little and publish

[kainwei]

I have met this problem today as well. Difference is I was updating keda-2.2.0 to keda-2.2.2.
Error message:

Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. 
Unable to continue with update: CustomResourceDefinition "scaledjobs.keda.sh" 
in namespace "" exists and cannot be imported into the current release: 
invalid ownership metadata; label validation error: 
missing key "app.kubernetes.io/managed-by": must be set to "Helm"; 
annotation validation error: missing key "meta.helm.sh/release-name": must be set to "keda"; 
annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "keda"  

I had searched on Internet, I found that is because the newer version's chart template crd(CustomResourceDefinition) already exist(highly likely from your installed versions before) in your k8s api resources and they had a conflict.
Actually, I don't know why this happened, because my another cluster has no this issue. So I searched for this again, I found a way to delete the api resources, then reinstall the keda to solve this problem.

Below is my steps what I have done(I used helm 3):

1. show your api resources and conflict resources:

$  kubectl api-resources|grep scaled
scaledjobs                                sj               keda.sh                        true         ScaledJob
scaledobjects                             so               keda.sh                        true         ScaledObject

2. list the template for each version(From this we can know keda 2.2.2 use name: scaledobjects.keda.sh(this is the conflict resource) which 2.2.0 not, that is why I can update my lower version before to 2.2.0 but cannot update the keda to 2.2.1 and 2.2.2. we can also know the difference between 2 versions, like the CustomResourceDefinition ):

$ helm template keda kedacore/keda --version 2.2.2|grep -E 'scaled|CustomResourceDefinition'
kind: CustomResourceDefinition
# Source: keda/templates/03-crd-scaledjobs.keda.sh.yaml
kind: CustomResourceDefinition
 name: scaledjobs.keda.sh
   plural: scaledjobs
   singular: scaledjob
       description: ScaledJob is the Schema for the scaledjobs API
# Source: keda/templates/04-crd-scaledobjects.keda.sh.yaml
kind: CustomResourceDefinition
 name: scaledobjects.keda.sh
   plural: scaledobjects
   singular: scaledobject
kind: CustomResourceDefinition
 - scaledjobs
 - scaledjobs/status
 - scaledobjects
 - scaledobjects/finalizers
 - scaledobjects/status
$ helm template keda kedacore/keda --version 2.2.0|grep -E 'scaled|CustomResourceDefinition'
 - scaledjobs
 - scaledjobs/status
 - scaledobjects
 - scaledobjects/finalizers
 - scaledobjects/status
$

3. Now it is our solution for this problem, delete the api resource and reinstall the keda:
   uninstall the current version

$ helm  uninstall keda -n keda 
release "keda" uninstalled
$ 

now, conflict resource still here:

$  kubectl api-resources|grep scaled
scaledjobs                                sj               keda.sh                        true         ScaledJob
scaledobjects                             so               keda.sh                        true         ScaledObject
$

Delete them by kubectl delete:

$ helm template keda kedacore/keda --version 2.2.1 | kubectl   delete -f -
customresourcedefinition.apiextensions.k8s.io "clustertriggerauthentications.keda.sh" deleted
customresourcedefinition.apiextensions.k8s.io "scaledjobs.keda.sh" deleted
customresourcedefinition.apiextensions.k8s.io "triggerauthentications.keda.sh" deleted
Error from server (NotFound): error when deleting "STDIN": serviceaccounts "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": customresourcedefinitions.apiextensions.k8s.io "scaledobjects.keda.sh" not found
Error from server (NotFound): error when deleting "STDIN": clusterroles.rbac.authorization.k8s.io "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": clusterroles.rbac.authorization.k8s.io "keda-operator-external-metrics-reader" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator:system:auth-delegator" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator-hpa-controller-external-metrics" not found
Error from server (NotFound): error when deleting "STDIN": rolebindings.rbac.authorization.k8s.io "keda-operator-auth-reader" not found
Error from server (NotFound): error when deleting "STDIN": services "keda-operator-metrics-apiserver" not found
Error from server (NotFound): error when deleting "STDIN": deployments.apps "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": deployments.apps "keda-operator-metrics-apiserver" not found
Error from server (NotFound): error when deleting "STDIN": apiservices.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io" not found
$ 

Run it again you would find all of the template resource was deleted:

$ helm template keda kedacore/keda --version 2.2.1 | kubectl  delete -f -
Error from server (NotFound): error when deleting "STDIN": serviceaccounts "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": customresourcedefinitions.apiextensions.k8s.io "clustertriggerauthentications.keda.sh" not found
Error from server (NotFound): error when deleting "STDIN": customresourcedefinitions.apiextensions.k8s.io "scaledjobs.keda.sh" not found
Error from server (NotFound): error when deleting "STDIN": customresourcedefinitions.apiextensions.k8s.io "scaledobjects.keda.sh" not found
Error from server (NotFound): error when deleting "STDIN": customresourcedefinitions.apiextensions.k8s.io "triggerauthentications.keda.sh" not found
Error from server (NotFound): error when deleting "STDIN": clusterroles.rbac.authorization.k8s.io "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": clusterroles.rbac.authorization.k8s.io "keda-operator-external-metrics-reader" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator:system:auth-delegator" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "keda-operator-hpa-controller-external-metrics" not found
Error from server (NotFound): error when deleting "STDIN": rolebindings.rbac.authorization.k8s.io "keda-operator-auth-reader" not found
Error from server (NotFound): error when deleting "STDIN": services "keda-operator-metrics-apiserver" not found
Error from server (NotFound): error when deleting "STDIN": deployments.apps "keda-operator" not found
Error from server (NotFound): error when deleting "STDIN": deployments.apps "keda-operator-metrics-apiserver" not found
Error from server (NotFound): error when deleting "STDIN": apiservices.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io" not found
$

Try use kubectl api-resources to check, we found it's clear:

$ kubectl api-resources |grep scaled
$ 

4 reinstall and successful:

$ helm install keda kedacore/keda -n keda
NAME: keda
LAST DEPLOYED: Tue May 11 11:33:38 2021
NAMESPACE: keda
STATUS: deployed
REVISION: 1
TEST SUITE: None
$ helm list -n keda
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART     	APP VERSION
keda	keda     	1       	2021-05-11 11:33:38.133763248 +0000 UTC	deployed	keda-2.2.2	2.2.0      
$ kubectl get pods -n keda
NAME                                              READY   STATUS    RESTARTS   AGE
keda-operator-546fc6c546-wfh98                    1/1     Running   0          7m24s
keda-operator-metrics-apiserver-df57db588-sl7m9   1/1     Running   0          7m24s
$ 

Problem I have met in this process:

1. each time the conflict resource might be not same, you need to try different version's template to find where is it:
   like:
   ...
   helm template keda kedacore/keda --version 2.2.0 | kubectl delete -f -
   helm template keda kedacore/keda --version 2.2.1 | kubectl delete -f -
   helm template keda kedacore/keda --version 2.2.2 | kubectl delete -f -
   ...

2. sometimes 'kubectl delete' shows that the resource was deleted, but it still there! I had tried to install the version which I can install without error(for my case it was 2.2.0) and uninstall process by '--no-hooks' (command: helm3 uninstall keda -n keda --no-hooks), then it worked. I don't really now the reason, because in official website, there is only one explaination: "prevent hooks from running during uninstallation"

About how to fix the conflict resource in helm, I referred this article:
https://stackoverflow.com/questions/59443834/helm-3-install-for-resouces-that-exist

At last, hope my article can be helpful to whom met this problem!

[tomkerkhove]

Thanks for the write-up, if I get it correctly it's just due to Helm 3, right?

[kainwei]

I am not sure what will happened in Helm2, but in my case, yes, I used Helm 3.