Helm post-install hooks are not triggered when used by helm_release and wait = true

[sherifabdlnaby]

Terraform, Provider, Kubernetes and Helm Versions

Terraform v0.14.6
+ provider registry.terraform.io/hashicorp/helm v2.0.2
+ provider registry.terraform.io/hashicorp/kubernetes v1.13.3

Helm chart with post-install hooks will nor be installed succsefully with wait = true, the provider behavior waits for installation to be READY, some charts won't get to the READY state without the post install step. eventually timeout is reached. Provider then exists *indicating a succsefully installation and will not install the post-install manfists.

Try install https://github.com/apache/airflow/tree/master/chart (you can download the directory instead of installing the entire repository).

You'll notice that the https://github.com/apache/airflow/blob/master/chart/templates/migrate-database-job.yaml is not installed by Helm when used by helm_release and use wait = true. Kindly note the annotations."helm.sh/hook": post-install,post-upgrade

Affected Resource(s)

• helm_release

Terraform Configuration Files

resource "helm_release" "chart" {
  name                 = "airflow"
  namespace            = "airflow"
  chart                = "./chart" # A local copy of `apache/airflow` chart.
  wait = true
}

Steps to Reproduce

1. terraform apply

Expected Behavior

Chart Installed Successfully

Actual Behavior

The chart runs and waits until timeout is passed, then it outputs that 1 resource created successfully, but when you check the chart it failed, and checking the Helm release, not all manifests are installed.

Important Factoids

None.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[sparkleGoat]

Appreciate this issue being documented. Just saved sometime by switching that wait configuration.

[Radu-Codrin]

Spent so much time investigating this and looking into Tiller logs, etc... manual install worked fine, but the hooks never run when deployed from my TF module.

[andormarkus]

Hi All,

To use helm_release to deploy airflow to a clean environment just set wait=false and delopment will work.

resource "helm_release" "airflow" {
  repository = "https://airflow-helm.github.io/charts"
  chart      = "airflow"
  name       = "airflow"
  version    = "8.0.9"
  namespace  = "airflow"
  wait       = false
}

I don't know what causes this bug, but I can tell what happens on the deployment and airflow side I hope it helps.
Manual installation steps:

1. helm runs the hooks and this will trigger the airflow db init which creates the database schema
2. helm deployments start which runs the init containers: check-db and wait-for-db-migrations
3. Init containers are successfull and the airflow containers

helm_release installation steps:

1. hooks wont run and airflow db init will not trigger, database schema will not be created
2. helm deployments start which runs the init containers: check-db and wait-for-db-migrations. check-db will pass however the wait-for-db-migrations will stuck in a "bootloop" with the following error message.

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 54, in check_migrations
    db.check_migrations(timeout=args.migration_wait_timeout)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 598, in check_migrations
    raise TimeoutError(f"There are still unapplied migrations after {ticker} seconds.")
TimeoutError: There are still unapplied migrations after 60 seconds.

This is cased by airflow check_migrations function which does the following check source_heads == db_heads. The 'db_heads' will be always empty because the database were newer initialised and schema was never created.

[RicoToothless]

face same issue with bitnami/kong too

Bitnami Kong also use post-install to create a job, but not show in events

kubectl -n <your_namespace> get events --sort-by='{.lastTimestamp}'

and thx @andormarkus workaround solution.

[cdibble]

I faced this issue as well and it took a minute to realize that it was my setting the --atomic flag in the helm command that was causing the problem- because setting --atomic apparently also sets --wait internally.

[RickRen7575]

I also was seeing this issue!

[yamen23ali]

I have the same issue with my own charts. If i remove the hook it works fine. otherwise, helm_release just ignore it.

[ffais]

Same problem here on my own chart!

[stargriv]

The same problem with 2.8.0 and CloudBees sidecar helm chart:

resource "helm_release" "sidecar" {
  chart      = "cloudbees-sidecar-injector"
  repository = "https://charts.cloudbees.com/public/cloudbees"
  version    = var.cbci_sidecar_chart_version
  name       = "cloudbees3-injector"
  namespace  = var.cbci_sidecar_namespace
  create_namespace = true
}

[BBBmau]

Got to replicate the install of charts that involve post-install hooks and found that with helm CLI that the chart will not install if --wait is set. Not much can be done in the provider since this is the expected behavior in helm when attempting to install charts that involve hooks, making sure that wait = false is important.

[mfjackson]

If anyone is coming here from data-on-eks self-managed-airflow, is testing locally, and already has wait = false in their airflow_helm_config, the following steps seemed to work for me (given by the issue submission template):

1. Remove the local .terraform directory: rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

This allowed me to successfully deploy updates using helm_release.

[jeremiahlee]

This bug also affects the official Mastodon Helm chart.

[LukaGiorgadze]

Yeh, unfortunately the only way to avoid this is:

resource "helm_release" "release-name" {
  wait   = false
  atomic = false
  ...
}

[Vedrillan]

If you still want to use the wait feature, airflow has actually documented how to do so:
https://airflow.apache.org/docs/helm-chart/stable/index.html#installing-the-chart-with-argo-cd-flux-rancher-or-terraform

[github-actions]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!