helm chart CRD installation first when installed as dependency #226

agnewp · 2022-01-10T18:59:33Z

A clear and concise description of what the bug is.

Expected Behavior

Keda can be installed from its chart as a dependency of another custom chart that uses Keda CRDs

Actual Behavior

Helm expects that CRDs need to be installed BEFORE other resources that potentially use these CRDS. when KEDA is a dependent chart of another one that attempts to use KEDA CRDs, the helm installation fails.

Steps to Reproduce the Problem

make a simple chart that depends on the keda chart. this simple chart should have a keda CRD in it at minimum, like a TriggerAuthentication object for example
use helm to attempt installation of the top-level custom chart
watch in horror as you get this error:

installation failed: unable to build kubernetes objects from release manifest: unable to recognize \"\": no matches for kind \"TriggerAuthentication\" in version \"keda.sh/v1alpha1\"

Specifications

KEDA Version: 2.5.1
Platform & Version:
Kubernetes Version: 1.20 eks
Scaler(s): N/A

the good news is that you aren't the first people to run into this problem, i see someone made a crude attempt at enforcing installation order by making the file names look like linux init.d scripts in some kind of assumption that helm doesn't do things in parallel. check this out: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/

the fix is supper simple; put the crd definitions into a folder called 'crds' and boom! helm will install the CRD defs first every time!

The text was updated successfully, but these errors were encountered:

tomkerkhove · 2022-01-11T13:50:45Z

the fix is supper simple; put the crd definitions into a folder called 'crds' and boom! helm will install the CRD defs first every time!

This is what we used to have, but the CRDs were not being updated then. Helm does not provide a good story here.

agnewp · 2022-01-11T17:08:17Z

well then its a trade-off. We have experienced that the current behavior for a new installation is quite flakey. the helm install command actually misses installation of some of the CRDs some of the time. also is there a point to how the files are named? does this actually work as a way to define installation order?

zroubalik · 2022-01-11T17:37:04Z

@agnewp I can see that you are reporting KEDA version 1.5.1, but that error mentiones:

no matches for kind "TriggerAuthentication" in version "keda.sh/v1alpha1"

which is a version of a resource used on KEDA 2.X.

I definitely recommend you to use 2.x version, 1.x is deprecated.

agnewp · 2022-01-11T19:15:35Z

my mistake, i am using 2.5.1

tomkerkhove · 2022-01-11T21:16:54Z

well then its a trade-off. We have experienced that the current behavior for a new installation is quite flakey

The alternative approach will silently continue giving you incomplete CRDs.

However, I will take a look when I have time but we might want to use both approaches as a hack: crd/ for new installations and the current approach for updates.

tomkerkhove · 2022-01-11T21:27:39Z

Let me try the chart again tomorrow though because last time I checked I didn't have issues.

tomkerkhove · 2022-01-12T12:39:15Z

So what I did was:

Create a fresh cluster (using AKS)
Check CRDs:

~  k get crds
NAME                                             CREATED AT
volumesnapshotclasses.snapshot.storage.k8s.io    2022-01-12T12:24:40Z
volumesnapshotcontents.snapshot.storage.k8s.io   2022-01-12T12:24:40Z
volumesnapshots.snapshot.storage.k8s.io          2022-01-12T12:24:40Z

Install KEDA

~  helm install keda --version 2.5.1 --namespace keda-system kedacore/keda
NAME: keda
LAST DEPLOYED: Wed Jan 12 13:37:00 2022
NAMESPACE: keda-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

Check CRDs:

~  k get crds
NAME                                             CREATED AT
clustertriggerauthentications.keda.sh            2022-01-12T12:37:05Z
scaledjobs.keda.sh                               2022-01-12T12:37:05Z
scaledobjects.keda.sh                            2022-01-12T12:37:05Z
triggerauthentications.keda.sh                   2022-01-12T12:37:05Z
volumesnapshotclasses.snapshot.storage.k8s.io    2022-01-12T12:24:40Z
volumesnapshotcontents.snapshot.storage.k8s.io   2022-01-12T12:24:40Z
volumesnapshots.snapshot.storage.k8s.io          2022-01-12T12:24:40Z

So everything seems fine.

Can you provide some more information about your environment? Keeping in mind that you use it as a Helm dependency of course, I'll test that next.

tomkerkhove · 2022-01-12T12:51:16Z

Ok I can reproduce this:

~  helm install metachart .
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "TriggerAuthentication" in version "keda.sh/v1alpha1"

Will take a stab at a fix later this week/next week. (will do my best)

tomkerkhove · 2022-01-12T13:00:35Z

https://github.com/tomkerkhove/keda-helm-dependency

tomkerkhove · 2022-01-12T15:05:01Z

Workaround: Install KEDA outside of the current Helm chart.

agnewp · 2022-01-12T16:46:29Z

This is indeed what we did to work around the issue. however, even installing Keda stand-alone with the current chart, we ran into inconsistent issues with the chart failing to install all of the CRDs. very strange. i wonder if some of the CRDs depend on eachother?

tomkerkhove · 2022-01-13T07:11:45Z

We do not have knowledge about that issue. Can you still reproduce this with the latest version?

tomkerkhove · 2022-01-23T13:23:22Z

Would it help if there is a Helm chart with solely the CRDs that you can install first?

simonlsk · 2022-03-09T07:49:20Z

This is what we used to have, but the CRDs were not being updated then. Helm does not provide a good story here.

Is there an open issue for that in helm?

Would it help if there is a Helm chart with solely the CRDs that you can install first?

Do you mean that we would add two dependencies in the chart like this:

- name: keda
  version: 2.5.1
  repository: https://kedacore.github.io/charts
- name: keda-crds
  version: 2.5.1
  repository: https://kedacore.github.io/charts

I think that could work for me.

tomkerkhove · 2022-03-09T07:56:49Z

Is there an open issue for that in helm?

Yes, see above but link is helm/helm#10585

Do you mean that we would add two dependencies in the chart like this:
- name: keda
  version: 2.5.1
  repository: https://kedacore.github.io/charts
- name: keda-crds
  version: 2.5.1
  repository: https://kedacore.github.io/charts
I think that could work for me.

No, so you can deploy the CRDs before the application. Having it as two dependencies would have same issue.

toniopelo · 2022-09-12T20:05:25Z

I would add that for some of us that are using skaffold with helm, this is even worst.
Breaking this in two separate charts to first install the keda chart with all the crds and after install the other chart that uses these crds as stated above doesn't work (stated here: #226 (comment)).

skaffold dev works fine as it first install the keda chart and then the other chart.
But when uninstalling it fails as it first uninstall the keda chart with all the crds (they are in templates/, so uninstalled as any other resource) and then when trying to uninstall the other chart that rely on the keda crds that doesn't exist anymore, it fails to uninstall it.

Then I tried to extract the crds elsewhere to handle them separately (a command that will apply them before installing the rest via helm, easily integrated with the skaffold flow thanks to the hooks commands).
There it would theoretically work and allow me to put back all the chart under one roof.
But it doesn't work because the keda chart include the crds under the template/ folder and these are conflicting with crds I already installed.

Having one chart keda-crds and another one keda without the crds like proposed by @tomkerkhove (#226 (comment)) would solve my problem as I could only install the keda one and avoid conflicts.
For now I'm stuck with unpacking the keda chart and removing the crds myself, this is a bit annoying, but this works.
This might be best of both world as you get to handle the keda-crds chart how you wish with the upgrade possibility and still allow the ones that would like to handle crds their way to use your keda chart.

tomkerkhove · 2022-09-13T05:34:58Z

Thanks for bumping this thread again, unfortunately no change on the Helm side.

Having one chart keda-crds and another one keda without the crds like proposed by @tomkerkhove Tom Kerkhove FTE (#226 (comment)) would solve my problem as I could only install the keda one and avoid conflicts

If we do this, then you would be unblocked if I get it correctly?

toniopelo · 2022-09-13T09:30:37Z

Hi @tomkerkhove, thanks for the fast reply :)!

unfortunately no change on the Helm side.

Yeah, the Helm implementation is really not ideal...

If we do this, then you would be unblocked if I get it correctly?

Yes that's right. I would be unblocked in my case, and I think that having both charts separated would likely solve other cases as well.

you can still have them both (nothing change)
you can handle the crds in another way that suits your needs and still use the main chart without risking conflicts.

tomkerkhove · 2022-09-13T09:36:58Z

What are your thoughts on this @kedacore/keda-maintainers? I think if we do this we should do the following:

Introduce keda-crd Helm chart with solely our CRDs
Introduce keda-runtime Helm chart with everything, except for our CRDs
Continue maintaining keda Helm chart, but use keda-crd & keda-runtime as dependencies.

Thoughts?

toniopelo · 2022-09-20T15:03:14Z

@tomkerkhove I just realised that you can use the following in the values.yaml:

crds:
  install: false

This will prevent crds from being installed on the current chart.
So I can use the chart as-is without any conflict and still installing the crds my way.
I don't think any action needs to be taken on your side then :).

tomkerkhove · 2022-09-21T11:32:40Z

Sounds good, thanks for letting us know!

brianV · 2022-12-23T20:47:01Z

I just ran into the same issue.

I ended up taking the CRDs from the 2.8.2.yaml file and installing them. I then added the helm annotations and labels. Following that, the helm chart installed correctly, and I know the removal of the CRDs will be managed by helm.

Adds autoscalers for the inbox-listener and stream service. The inbox listener autoscaler relies on KEDA. The stream service autoscaler is a standard HPA. KEDA has some unfortunate friction with helm. It cannot be added as a dependency to the chart and installed in a reasonable way, because it relies on CRDs, which helm will not be responsible for updating or installing in order. Based on conversation in kedacore/charts#226, there are two alternative approaches open to us: * We can instruct users to install KEDA on their cluster prior to installing our deployment. The command for this is, helm install keda --version 2.9.1 --namespace keda kedacore/keda --create-namespace * We can vendor KEDA's CRDs and put them into our chart, under a crd directory. This should enable a single-command install, but comes with the downsides that, 1. Helm will not update or manage the keda CRDs for us 2. If users already have KEDA installed (certainly possible) we will encounter a conflict. Mainly due to the second item, this PR takes the first approach.

wmagon · 2023-02-15T16:26:33Z

[TL;DR]
even though CRDs are not clear for Helm, the separate /crds location seems to be a standard, prepared to provide compatibility with different approaches and giving some choice.
For our project, where we use Terraform with helm_release resource, it's extremely useful to detach CRDs from Helm Charts,that provides us flexibility with recreating/destroying resources automated way many times, without removing CR objects. Detached CRDs are applied/updated with kubectl (in null_resource) before helm_release and it's convenient if that is separate single file, stored in different catalog, in Release assets, even in different repo would work, etc.

[Longer description]
I think it would still be very helpful if CRDs would have separate location (e.g. /crds). If not for skipping them with Helm flag, then for installing in separate step. Correct me if I'm wrong, but currently it seems to be tricky to install keda CRDs outside the Helm and not controlled by Helm (needed some kind of extracting from template the few .yaml files, generate manifest that could be applied after removing from it helm annotations/labels?). I think it's safer and quite common scenario, than someone would like to avoid removal of CRDs during Chart deletion and wants to pre-install them first.

Many projects are struggling with similar Helm challenges, but I guess I've everywhere seen recommendations and solutions that allows extracting CRDs from the Chart, e.g.:

Kong
They provide some explanation:
https://github.com/Kong/charts/tree/kong-2.16.4/charts/kong#crd-management
and CRDs can be applied with kubectl from one .yaml file located in /crds
cert-manager
with separate /crds location
https://github.com/cert-manager/cert-manager/tree/master/deploy/crds
and single .yaml file in Releases assets, that can be referenced with kubectl as officially documented
https://cert-manager.io/docs/installation/helm/#option-1-installing-crds-with-kubectl
https://github.com/cert-manager/cert-manager/blob/v1.11.0/deploy/charts/cert-manager/README.template.md
aws-load-balancer-controller
with single .yaml file in separate location and instructions for kubectl
https://github.com/kubernetes-sigs/aws-load-balancer-controller/tree/v2.4.6/helm/aws-load-balancer-controller
AWS VPC CNI (amazon-vpc-cni-k8s)
until recently, in instructions about adopting pre-made objects into Helm, there was clear suggestion to not do that for CRDs
https://github.com/aws/amazon-vpc-cni-k8s/tree/v1.12.0/charts/aws-vpc-cni#adopting-the-existing-aws-node-resources-in-an-eks-cluster
now the separate /crds location has been prepared, with commit named "Move CRD resources to a separate folder as per helm standard"

tomkerkhove · 2023-03-07T06:51:42Z

Correct me if I'm wrong, but currently it seems to be tricky to install keda CRDs outside the Helm and not controlled by Helm (needed some kind of extracting from template the few .yaml files, generate manifest that could be applied after removing from it helm annotations/labels?).

This is not 100% correct - If you use Helm, it will install/update them. For non-Helm scenarios we offer them as YAML in our GitHub release.

I'm happy to adapt and align with the /crd pattern, but does it actually install them? That is not clear to me.

wmagon · 2023-03-07T13:52:01Z

I meant Helm scenario and the whole point is to have any option to use Helm, but without CRDs and easy way to install CRDs separately.
Currently even if we would use crds.install = false, there is no simple way to find only CRDs manifests, as they are always part of the gathered config. Also mentioned yaml manifests gives only option to apply everything or nothing.
All examples mentioned above gives the option for that separation. Some yaml on Github releases with just CRDs (like cert-manager), or /crd with such yaml, both would work and provide that helpful flexibility.

tomkerkhove · 2023-03-08T10:15:25Z

So having a YAML file with just CRDs in the releases would help?

wmagon · 2023-03-08T11:25:20Z

In our case yes. We would have an easy way to install/maintain only CRDs first and then Helm Chart without CRDs.

daodennis-ri · 2023-08-31T02:38:53Z

What is the problem if you don't skip CRDs with the installation of the main chart? I'm trying to understand why skipCRDs is a thing

wmagon · 2023-08-31T09:22:25Z

@daodennis-ri
In case of this chart the CRDs will be installed wrong way, with helm annotations, tracked by helm and removed during helm uninstall, what might remove Custom Resources related to those CRDs. That's why Helm decided to go with safer approach: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/.

Usually it's not big difference whether CRDs will be installed first time with Helm or without (e.g. kubectl) if can be later easily maintained (updated) outside and if are not tracked by Helm later on , something like Methond 1 in Helm docs. The Helm idea is to allow such installation (or skip) using manifests in /crds where "These CRDs are not templated".

With KEDA chart the issue and main difference is that /crds files are "templated" and I think /crds location doesn't work as designed and CRDs are annotated/labeled and tracked by helm. There is an option to disable CRDs with crds.install though. To install CRDs outside helm, instead of using /crds templated files and unpinning helm tracking by removing annotations, some manifests located in kedacore/keda should allow doing it much easier.

edeustua · 2023-09-05T02:45:43Z

I'm having a similar problem. I'm trying to install cloudnative-pg's helm chart as a dependency (set in Chart.yaml) and helm is not able to install the corresponding CRDs before attempting to install my chart.

sidharthpunathil · 2023-11-20T20:11:41Z

+1 Same Problem

edeustua · 2024-01-15T15:28:29Z

After fiddling with this for a while, I've reached the conclusion that the concept of Helm subcharts is fundamentally flawed. This is specially true for CRDs, or charts containing non-namespaced resources.

The solution we have arrived at is the same as the one proposed by most Helm charts containing CRDs. Have them as a separate Helm installation, decoupled for your target chart.

rajpopat27 · 2024-09-12T12:27:49Z

So what is the recommended approach should we install keda out of the main helm chart and not add it as a dependency? Because I am still facing this issue where it fails saying CRD's doesn't exist

ana-cafemedia · 2024-09-25T06:01:27Z

@tomkerkhove Any update on your keda-crd chart? We are encountering the same issue and would like to be able to use that.

agnewp added the bug Something isn't working label Jan 10, 2022

tomkerkhove added the help wanted Extra attention is needed label Jan 11, 2022

tomkerkhove changed the title ~~helm chart CRD installation first~~ helm chart CRD installation first when installed as dependency Jan 12, 2022

tomkerkhove mentioned this issue Jan 19, 2022

Managing CRDs & chart dependendies helm/helm#10585

Closed

akshay196-rafay mentioned this issue Apr 28, 2022

Contour CRDs are installing even when contour dependency is disabled paralus/helm-charts#13

Open

wkalt mentioned this issue Jan 11, 2023

Introduce autoscalers for inbox listener and stream service foxglove/helm-charts#18

Closed

JorTurFer modified the milestones: KEDA Core v2.10, KEDA Core vNext Mar 9, 2023

qaribhaider mentioned this issue Apr 2, 2023

No matches for kind "ScaledObject" mguay22/nestjs-metrics#1

Open

tomkerkhove mentioned this issue May 11, 2023

feat: Support extra kubernetes manifests #395

Merged

3 tasks

tomkerkhove mentioned this issue May 16, 2024

Provide a helm chart just for the CRDs like karpenter does kedacore/keda#5575

Open

tomkerkhove mentioned this issue Sep 3, 2024

Add keda-crd helm chart #675

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

helm chart CRD installation first when installed as dependency #226

helm chart CRD installation first when installed as dependency #226

agnewp commented Jan 10, 2022 •

edited

Loading

tomkerkhove commented Jan 11, 2022

agnewp commented Jan 11, 2022

zroubalik commented Jan 11, 2022

agnewp commented Jan 11, 2022

tomkerkhove commented Jan 11, 2022

tomkerkhove commented Jan 11, 2022

tomkerkhove commented Jan 12, 2022

tomkerkhove commented Jan 12, 2022

tomkerkhove commented Jan 12, 2022

tomkerkhove commented Jan 12, 2022

agnewp commented Jan 12, 2022

tomkerkhove commented Jan 13, 2022

tomkerkhove commented Jan 23, 2022

simonlsk commented Mar 9, 2022

tomkerkhove commented Mar 9, 2022

toniopelo commented Sep 12, 2022

tomkerkhove commented Sep 13, 2022

toniopelo commented Sep 13, 2022

tomkerkhove commented Sep 13, 2022 •

edited

Loading

toniopelo commented Sep 20, 2022

tomkerkhove commented Sep 21, 2022

brianV commented Dec 23, 2022

wmagon commented Feb 15, 2023

tomkerkhove commented Mar 7, 2023

wmagon commented Mar 7, 2023

tomkerkhove commented Mar 8, 2023

wmagon commented Mar 8, 2023

daodennis-ri commented Aug 31, 2023

wmagon commented Aug 31, 2023

edeustua commented Sep 5, 2023

sidharthpunathil commented Nov 20, 2023

edeustua commented Jan 15, 2024

rajpopat27 commented Sep 12, 2024

ana-cafemedia commented Sep 25, 2024

helm chart CRD installation first when installed as dependency #226

helm chart CRD installation first when installed as dependency #226

Comments

agnewp commented Jan 10, 2022 • edited Loading

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

tomkerkhove commented Jan 11, 2022

agnewp commented Jan 11, 2022

zroubalik commented Jan 11, 2022

agnewp commented Jan 11, 2022

tomkerkhove commented Jan 11, 2022

tomkerkhove commented Jan 11, 2022

tomkerkhove commented Jan 12, 2022

tomkerkhove commented Jan 12, 2022

tomkerkhove commented Jan 12, 2022

tomkerkhove commented Jan 12, 2022

agnewp commented Jan 12, 2022

tomkerkhove commented Jan 13, 2022

tomkerkhove commented Jan 23, 2022

simonlsk commented Mar 9, 2022

tomkerkhove commented Mar 9, 2022

toniopelo commented Sep 12, 2022

tomkerkhove commented Sep 13, 2022

toniopelo commented Sep 13, 2022

tomkerkhove commented Sep 13, 2022 • edited Loading

toniopelo commented Sep 20, 2022

tomkerkhove commented Sep 21, 2022

brianV commented Dec 23, 2022

wmagon commented Feb 15, 2023

tomkerkhove commented Mar 7, 2023

wmagon commented Mar 7, 2023

tomkerkhove commented Mar 8, 2023

wmagon commented Mar 8, 2023

daodennis-ri commented Aug 31, 2023

wmagon commented Aug 31, 2023

edeustua commented Sep 5, 2023

sidharthpunathil commented Nov 20, 2023

edeustua commented Jan 15, 2024

rajpopat27 commented Sep 12, 2024

ana-cafemedia commented Sep 25, 2024

agnewp commented Jan 10, 2022 •

edited

Loading

tomkerkhove commented Sep 13, 2022 •

edited

Loading