datahub-elasticsearch-setup-job fails to start in GKE

[Keqi]

Describe the bug
I am following the instructions from the documentation on how to deploy DataHub in GCP. I successfully managed to set up the Kubernetes cluster as well as install the prerequisites via helm. When I try to install helm install datahub datahub/datahub, I am running into errors when datahub-elasticsearch-setup-job spins up. Here's how to the cluster looks like (it seems to be retrying to install elasticsearch a couple of times)

NAME                                               READY   STATUS    RESTARTS        AGE
datahub-elasticsearch-setup-job-c8j6w              0/1     Error     0               4m22s
datahub-elasticsearch-setup-job-cbsp2              0/1     Error     0               4m26s
datahub-elasticsearch-setup-job-l8kmr              0/1     Error     0               3m52s
datahub-elasticsearch-setup-job-lgvpr              0/1     Error     0               2m32s
datahub-elasticsearch-setup-job-q2hx4              0/1     Error     0               4m12s
elasticsearch-master-0                             1/1     Running   0               6m44s
elasticsearch-master-1                             1/1     Running   0               6m43s
elasticsearch-master-2                             1/1     Running   0               6m43s
prerequisites-cp-schema-registry-cf79bfccf-gh456   2/2     Running   1 (5m54s ago)   6m44s
prerequisites-kafka-0                              1/1     Running   0               6m44s
prerequisites-mysql-0                              1/1     Running   0               6m44s
prerequisites-neo4j-community-0                    1/1     Running   0               6m44s
prerequisites-zookeeper-0                          1/1     Running   0               6m44s

And here's the log from failed pods:

2022/09/27 06:40:14 Waiting for: http://elasticsearch-master:9200
2022/09/27 06:40:14 Received 200 from http://elasticsearch-master:9200
Going to use protocol: http
Going to use default elastic headers
Create datahub_usage_event if needed against Elasticsearch at elasticsearch-master:9200
Going to use index prefix::
curl: option -k http://elasticsearch-master:9200/_ilm/policy/datahub_usage_event_policy: is unknown
curl: try 'curl --help' or 'curl --manual' for more information
/create-indices.sh: line 41: [: -eq: unary operator expected
/create-indices.sh: line 45: [: -eq: unary operator expected
/create-indices.sh: line 47: [: -eq: unary operator expected
Policy GET response code is
Got response code  while creating policy so exiting.
2022/09/27 06:40:14 Command exited with error: exit status 1

I was trying to search for a similar problem in your closed GH issues or all over the internet but I couldn't find the solution. Is this about missing curl installation or perhaps about that unary operator? I would really appreciate helping me with this one.

[Keqi]

I managed to resolve this issue by downgrading the version of elasticsearchSetupJob to 0.8.44 in the values.yaml. It seems like this commit might have affected the whole setup: acryldata/datahub-helm@6bd69fe.

Closing this issue as this is resolved. Please update your configuration setup so it can be installed without errors :)

[jinlintt]

I encountered the same problem during upgrading to v0.8.45.
We should leave this open as this is a bug in the v0.8.45 implementation of elasticsearch-setup-job.

FYI @jjoyce0510 @pedro93

[sachinwadhwa]

I agree, this should not be closed as its a bug in v0.8.45. Same issue encountered in Azure.

[pedro93]

Hello folks,

For additional context, what version of DataHub Helm charts are you using?

[jinlintt]

@pedro93 I used datahub-0.2.106 through helm repo update.
https://github.com/acryldata/datahub-helm/releases/tag/datahub-0.2.106

[pedro93]

Put up a PR to fix the issue, will try to get it in as soon as possible.
Sorry for the inconvenience everyone!

[jinlintt]

@pedro93 @anshbansal
How should we take in the fix? Should the docker image tag be updated?
v.0.8.45 is still from six days ago.
https://hub.docker.com/r/linkedin/datahub-elasticsearch-setup/tags?page=1&name=v0.8.45

Or should we just stick with v.0.8.44 for the elastic search setup job until the next release?

[anshbansal]

@jinlintt Stick with v.0.8.44 for now for elastic setup job. When new release is made you would be able to get the fix in.

[jinlintt]

@anshbansal got it. Thank you!

[pedro93]

If you need v0.8.45 features you can set the image tag for elasticsearch_setup_job to be: 596d484