Permissions on elasticsearch.keystore prevent Elasticsearch from starting

[pemontto]

From 6.2, and perhaps earlier, it appears the elasticsearch.keystore file is created even if X-Pack security isn't enabled. This then prevents Elasticsearch from starting up

Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/elasticsearch/0/elasticsearch.keystore

from tasks/xpack/elasticsearch-xpack.yml

#Make sure elasticsearch.keystore has correct Permissions
- name: Set elasticsearch.keystore Permissions
  become: yes
  file: state=file path={{ conf_dir }}/elasticsearch.keystore owner={{ es_user }} group={{ es_group }}
  when: es_enable_xpack and "security" in es_xpack_features and (es_version | version_compare('6.0.0', '>'))

I'd suggest removing "security" in es_xpack_features.
I can create a PR to relax the conditional, and without testing back on 6.x versions ignore errors. Though based on this and other issues, it seems we need some more automated tests around X-Pack features in enabled/disabled states. I will try to look into this when I get some time.

[pemontto]

Confirmed the issues exists on 6.1.3 also.

[deppfx]

I am running into the same issue with Chef. Were you ever able to get this fixed? I know they set a GUID on ES_HOME. Probably setting GUID on /etc/elasticsearch should do the trick?

[shorsher]

Is there a current work-around so I can start my elasticsearch manually until this is fixed?

[pmb311]

@shorsher

sudo chown root:elasticsearch <ES_PATH_CONF>/elasticsearch.keystore should work.

[jasontedor]

Yes, what is needed here is for chmod g+s ${ES_PATH_CONF}/elasticsearch after chown -R root:elasticsearch ${ES_PATH_CONF}. The setgid ensures that when the keystore is created it has the right ownership (root:elasticsearch).

[sufiyanghori]

Confirming that the issue still exists in 6.3.2, when I tried to start ES after upgrading from 5.6.
In my case, there was no write permission to /etc/elasticsearch, had to explicitly set both guid and w,

chmod g+ws /etc/elasticsearch/

[kzalewski]

This entire problem is being caused by an incorrect mixing of static, read-only configuration (elasticsearch.yml, jvm.options, log4j2.properties) with writable, run-time files (elasticsearch.keystore).

The /etc directory, and its subdirectories, should not need to be writable by non-privileged users. That's what /var and /tmp are for.

My contention is that elasticsearch.keystore is being improperly stored in /etc/elasticsearch/, when it should really be in something like /var/lib/elasticsearch/, which would be owned by the elasticsearch user. Unfortunately, the location of elasticsearch.keystore does not seem to be configurable.

[klarose]

I agree with @kzalewski . I am running elasticsearch in kubernetes, providing its configuration via a ConfigMap. ConfigMaps are read-only, meaning that if I mount it as a directory, and point elasticsearch to that directory as the config location, elasticsearch fails to work. This is a very simple deployment method, and I would really expect it to work.

At the very least, there should be a method to override the keystore location. But, it should probably be completely decoupled from static config by default.

[JannikZed]

Same for me .. I have no idea how I should run elasticsearch with configuration files as kubernetes config maps.. right now it's impossible as the directory needs to be writable for elasticsearch.keystore .. has anybody a working workaround for it?

[klarose]

It's possible. Just a pain. You need to mount the individual files. You can see an example in the elasticsearch helm chart, in fact:

https://github.com/helm/charts/tree/master/incubator/elasticsearch/templates

Take a look at configmap.yaml and client-deployment.yaml.

The configmap has an elasticsearch.yml defined, and the client-demployment mounts it directly onto the expected elasticsearch.yml in the container's filesystem.

[JannikZed]

Ahh thats perfect! Thank you for the hint. That’s okay for me, it worked!

…

________________________________ Von: Kyle Larose <[email protected]> Gesendet: Dienstag, Oktober 9, 2018 8:34 PM An: elastic/ansible-elasticsearch Cc: JZinkl; Comment Betreff: Re: [elastic/ansible-elasticsearch] Permissions on elasticsearch.keystore prevent Elasticsearch from starting (#430) It's possible. Just a pain. You need to mount the individual files. You can see an example in the elasticsearch helm chart, in fact: https://github.com/helm/charts/tree/master/incubator/elasticsearch/templates Take a look at configmap.yaml and client-deployment.yaml. The configmap has an elasticsearch.yml defined, and the client-demployment mounts it directly onto the expected elasticsearch.yml in the container's filesystem. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#430 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHLXHqxoBFC14MDdw-M4MD1w8arDlVusks5ujOwugaJpZM4SMv8_>.

[jmlrt]

Related to #560 (comment)

[kzalewski]

Hi Julien,

I noticed that you closed this issue. I just wanted to mention that we upgraded to ES 7.3.2 today, and the keystore file permission issue is still present. In fact, it's worse, because ES is trying to write a keystore.tmp file in the /etc/elasticsearch.d/ directory, and that directory does not allow write access. (Previously, I had to allow write access to only the file /etc/elasticsearch.d/elasticsearch.keystore as a workaround to this bug.)

If this permission issue is fixed, can you tell me in which version it is fixed?

[jmlrt]

Hi @kzalewski,
This has been fixed in https://github.com/elastic/ansible-elasticsearch/pull/609/files#diff-36a4a2a2e38012ed8ae3394c87bafa16L23-R11 so if you are using the master branch of this ansible role you shouldn't have anymore permissions issues on elasticsearch.keystore.

[kzalewski]

Thanks for the quick response @jmlrt. I'm not using Ansible to deploy ES. I simply download the Linux tar.gz package and install it that way. I'm guessing others, like me, were brought to this issue when we all had permissions issues with the ES startup, and searched for a solution on Google. This is why I asked which version of ES has the "fix", because I just upgraded from 6.7.2 to 7.3.2, and the permission problems still persist. I continue to maintain that /etc should not need non-root writable permissions, and that the keystore file should be located in a writable volume like /var.

[jmlrt]

Well, this repository is only using Ansible to deploy Elasticsearch. We had an issue because this role was changing some default Elasticsearch files permissions and we removed this code to keep default permission when we can.

If you still have a problem not related to Ansible role, can you open an issue to https://github.com/elastic/elasticsearch/ or create a topic on https://discuss.elastic.co/c/elasticsearch so it can be fixed in Elasticsearch code?

[lhoncorty]

chown elasticsearch:elasticsearch elasticsearch.keystore
Solved my issue.