bitnami/elasticsearch chart not emitting metrics with latest 7.13 release

[gkulkarni32]

Which chart:
bitnami/elasticsearch

Describe the bug
There was a bug in the elasticsearch upcoming prometheus exporter image of the latest release 7.13 in which some of the critical metrics were not being emitted. Below is the reference to the bug: prometheus-community/elasticsearch_exporter#419
This bug is fixed in the latest exporter image, we need to have the same fix in the latest Bitnami elasticsearch chart.

To Reproduce
Steps to reproduce the behavior:

1. Install the latest Elasticsearch chart with metrics enabled.
2. Look for the metrics being emitted by the exporter. some critical metrics like elasticsearch.jvm.memory.* are not being emited.
3. Checked the latest release of the Prometheus exporter 'quay.io/prometheuscommunity/elasticsearch-exporter:latest' and tested this chart with this image. It emits all the missing metrics.
   Merge request for this: Release 1.2.0 prometheus-community/elasticsearch_exporter#431

Expected behavior
All the missing metrics should be emitted by the exporter. one example is: elasticsearch.jvm.memory.*

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"dirty", GoVersion:"go1.15.5"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T21:51:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.14+vmware.1", GitCommit:"bbf52cd6cf83a50507e943b717ed321d383a37b5", GitTreeState:"clean", BuildDate:"2020-08-19T23:48:38Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Additional context
Add any other context about the problem here.

[carrodher]

Hi, thanks for letting us know.

We already released the new version for the prometheus-exporter, see DockerHub.

The next step is to update the elasticsearch-exporter image bundled in the ES Helm Chart to use the new version, see

charts/bitnami/elasticsearch/values.yaml

Line 1127 in da9f873

tag: 1.1.0-debian-10-r456

All the bundled images are automatically updated in each Helm Chart release, I just triggered a new release of the ES chart. Once tested on our internal pipeline, a new version should be pushed to GitHub and the Chart repositories containing the latest bitnami/elasticsearch-exporter version.

Once released the new ES Helm Chart, the last step is to release those charts using ES as a subchart, in this case, we have two Helm Charts depending on ES: dataplatform-bp2 and magento.

I just triggered this process, everything should be ready soon.

[gkulkarni32]

Thank you @carrodher for the update. Will wait for the ES and dataplatform-bp2 chart release.

[ebuildy]

@carrodher not sure this will work with elasticsearch-exporter 1.2.0 (see prometheus-community/elasticsearch_exporter#419).

Btw, what is the bitnami/elasticsearch-exporter image? thanks you.

[gkulkarni32]

Hi @carrodher , it still doesnt seem to emit the metrics for jvm. Can you please check this from your side?

[carrodher]

Hi @carrodher, it still doesn't seem to emit the metrics for JVM. Can you please check this from your side?

Hi, the new Elasticsearch version (15.3.1) was released yesterday around 4 PM CET, some hours after your comment so probably you tested the previous version that doesn't contain the following changes 2354722. Please note once the code is released in GitHub (previous commit) some time may pass until the chart is published in the repository since some internal processes are required.

@carrodher not sure this will work with elasticsearch-exporter 1.2.0 (see prometheus-community/elasticsearch_exporter#419).

In this case, since it is an upstream issue we can't do anything else apart from using the latest version of the prometheus-exporter container image and the elasticsearch image. Once solved the issue upstream we will release a new version of the chart including the latest versions of all the containers bundled in the chart but at this moment we can't modify the source code or binaries for the exporter nor elasticsearch itself.

You can also overwrite the image by using the image.tag parameter from the values.yaml to use a previous version that doesn't contain the bug introduced in the latest version.

Btw, what is the bitnami/elasticsearch-exporter image? thank you.

Here you can find the bitnami/elasticsearch-exporter source code as well as some explanation in the README.

[ebuildy]

Thanks you, I got it!

[carrodher]

The new version of dataplatform-bp2 containing the latest version of ES was already released (see above PR with the changes):

$ helm repo update && helm search repo bitnami/dataplatform-bp2
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
NAME                    	CHART VERSION	APP VERSION	DESCRIPTION
bitnami/dataplatform-bp2	3.0.1        	1.0.0      	OCTO Data platform Kafka-Spark-Elasticsearch He...

If the issue is not solved with those changes we need to wait until a new version of Elasticsearch or elasticsearch-exporter appears to fix the issue, but as it is something related to the upstream project from Bitnami we can't do anything else apart from using the latest version available at this moment.

I am closing the issue, but feel free to reopen it if you found that a new version is released and the charts are not updated automatically or if there is a valid workaround provided by the upstream developers we can apply to the chart.

[gkulkarni32]

Hi @carrodher
There is a fix that's gone in the latest exporter release which fixes the metrics. we tested with the exporter image.
prometheus-community/elasticsearch_exporter#439

Could you please let us know once this is added to the elasticsesarch and dataplatform-bp2 chart?

Thank you
Geeta

[carrodher]

Hi! The PR was merged in their source code but they didn’t release a new version yet, being the latest one 1.2.0 that is the one already bundled in our chart. We need to wait until a new version appears as released at https://github.com/prometheus-community/elasticsearch_exporter/tags

I'm reopening the ticket just in case, but we will need to wait until the es_exporter team release a new version

[adammike]

They haven't released a new version, but :master tag contains the fix and works well.

[carrodher]

Yes, the PR was merged in master so if it is working fine there is good news, that means once a new release is available everything should work fine as well.

We can't use the code from master since each change should be associated with a new version, it is needed to track the changes in a proper way, having a changelog of what is included in each version and also bump the version depending on the changes, it's not the same a major change than a patch change, those are the reason why we need to wait until the upstream developers officially release a new version following their own workflow for that.

[gkulkarni32]

@carrodher FYI Elasticsearch exporter 1.2.1 is released now.
prometheus-community/elasticsearch_exporter#439

[carrodher]

Hi @gkulkarni32, thanks for letting us know. The Bitnami automated pipeline already detected the new version and triggered the release process:

• First, it is needed to release the new version for the bitnami/elasticsearch-exporter container image. This is already done, if you take a look at DockerHub, a new tag (1.2.1) was released ✅
• The second step is to release a new bitnami/elasticsearch Helm Chart bundling the latest bitnami/elasticsearch-exporter container image. This is currently in progress, as you can see here (3d159ec) the ES Helm Chart was already updated in terms of code, now the process to push the new version to the Helm registry is running (this step can take some time since there are several tests & process involved) 🕐
• Finally, it is needed to release a new version of the Helm Charts using bitnami/elasticsearch as a dependency, in this case, bitnami/magento and bitnami/dataplatform-bp2. But this is done by running helm dep update so the new version of the bitnami/elasticsearch Helm Chart should be present in the Helm registry 🕐

At this moment the release pipeline is in the middle of the second step, but just to let you know that we are aware of the new version and the release is following the regular workflow.

[gkulkarni32]

Thanks @carrodher for the update. Will wait for the release of both elasticsearch and dataplatform-bp2 chart release.

[carrodher]

A new elasticsearch version (15.7.2) was released

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm search repo bitnami/elastic
NAME                 	CHART VERSION	APP VERSION	DESCRIPTION
bitnami/elasticsearch	15.7.2       	7.13.2     	A highly scalable open-source full-text search ...

$ helm fetch bitnami/elasticsearch --untar

$ ag ' bitnami/elasticsearch-exporter' elasticsearch/values.yaml -C 1
1169-    registry: docker.io
1170:    repository: bitnami/elasticsearch-exporter
1171-    tag: 1.2.1-debian-10-r0

As you can see per the last command, the bundled version of the exporter is 1.2.1

Now it is time to trigger a release of dataplatform-bp2 to include this ES version

[carrodher]

The above PR bumps all the dependencies bundled in the dataplatform-bp2 Helm chart; the ES version bundled is 15.7.2, which is the one with the latest version of the exporter.

This version is already released

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm search repo bitnami/dataplatform-bp2
NAME                    	CHART VERSION	APP VERSION	DESCRIPTION
bitnami/dataplatform-bp2	3.0.2        	1.0.0      	OCTO Data platform Kafka-Spark-Elasticsearch He...

Summarizing:

• bitnami/elasticsearch 15.7.2 includes elasticsearch-exporter 1.2.1 container image
• bitnami/dataplatform-bp2 3.0.2 includes elasticsearch 15.7.2 as subchart

Feel free to reopen the issue if you find any issue with the latest versions of the charts, as previously if there is something not working because of the exporter itself, we will need to wait for a new version in the upstream project.

[gkulkarni32]

Thank you @carrodher for the update. We will test it out.