Skip to content
This repository has been archived by the owner on Jun 24, 2022. It is now read-only.

[Feature] 7.0 support #554

Closed
abraxxa opened this issue Apr 15, 2019 · 26 comments · Fixed by #558 or #573
Closed

[Feature] 7.0 support #554

abraxxa opened this issue Apr 15, 2019 · 26 comments · Fixed by #558 or #573
Assignees

Comments

@abraxxa
Copy link

abraxxa commented Apr 15, 2019

When will the playbook be ready for 7.0?

@wixaw
Copy link

wixaw commented Apr 25, 2019

We're waiting impatiently 👍

@jmlrt jmlrt self-assigned this Apr 26, 2019
@abraxxa
Copy link
Author

abraxxa commented May 2, 2019

Can we please get an update from Elastic on this issue?

@jmlrt
Copy link
Member

jmlrt commented May 2, 2019

Hi @abraxxa work is in progress to update ansible-elasticsearch to 7.x.
We should have a PR for it soon.

@abraxxa
Copy link
Author

abraxxa commented May 2, 2019

@jmlrt thanks for the update!

@abraxxa
Copy link
Author

abraxxa commented May 9, 2019

Great! Will you also do a release of the role?

@jmlrt
Copy link
Member

jmlrt commented May 9, 2019

@abraxxa The PR merge automatically closed your issue but yes we still need to do the release.

@jmlrt jmlrt reopened this May 9, 2019
@mponton
Copy link

mponton commented May 14, 2019

Bump. Is the 7.0 support official and soon to be "officially released"? I'm good to pull the latest version and use it but some of my colleagues would feel more comfortable with a tag and official release 😃

Thanks!

@hjst
Copy link

hjst commented May 16, 2019

Is there anything that we non-Elastic-co folks can do to help this release? The README says that to get started we should do:

ansible-galaxy install elastic.elasticsearch,7.0.1

But that obviously results in an error because 7.0.1 isn't tagged. Cloning master seems to work as expected, but the mismatch between the docs & the tagged release over a week after the merge is giving me pause. Is there an unseen testing process running that needs to complete?

@jmlrt
Copy link
Member

jmlrt commented May 16, 2019

Sorry about the delay for the release and mismatch between README file and tagged release.

We still have 2 PR to add before adding the release tag:

Unfortunately, I'm not available to work on it this week and should continue next week.

Meanwhile using the repository master branch is fine and tested.

@abraxxa
Copy link
Author

abraxxa commented May 20, 2019

The removal of multi-instance support is quite a large change!
Will it handle the changed directory paths?

@ghost
Copy link

ghost commented May 21, 2019

Why multi-instances support should be removed?
If so, it will be very terrible for me because I just have some servers with huge memory. I can not deploy only one instance per server which is really wasteful.

@jmlrt
Copy link
Member

jmlrt commented May 29, 2019

Multi-instances support was implemented when it was the only solution available to optimize usage of big servers with a lot of memory as well as allow cluster testing locally.

It came with a lot of drawbacks as it required to override Elastic official packages behavior by changing standard configuration path, data path and log path (even in single instance configuration) in this ansible role as in the puppet module we also provide.

One major point is the need to provide customized init scripts §(systemd or init depending on OS) instead of the ones provided by official packages.

This is bringing confusion to a lot of users and is responsible of a lot of the complexity and maintenance burden in our code.

Currently you are no more restricted to this only solution when needed to setup more than one instance of elasticsearch on the same host as Elastic is fully supporting Docker containers and is providing official Docker images.

We are now recommending Docker containers when there is some use-case which requires installing multiple elasticsearch instances in the same host.

That's why we are taking profit of Elasticsearch 7.0 major release to integrate this breaking change and remove multi-instances support.

This change should still allow you to change directory paths by overriding ansible variables as well as http and transport ports.

For existing multi-instances uses cases, the master branch is already working with elasticsearch 7.X and multi-instance support. You only need to use ansible-galaxy install git+https://github.com/elastic/ansible-elasticsearch.git,86c8e3c59bf2e34b62091cc766dd5f84df5fbef0 instead of ansible-galaxy install elastic.elasticsearch,7.0.1.

@wixaw
Copy link

wixaw commented Jun 4, 2019

Hello @jmlrt
Est ce que cela fonctionne pour 7.1 ? Car je souhaite me débarrasser de searchguard
Thanks

@jmlrt
Copy link
Member

jmlrt commented Jun 4, 2019

Hello @wixaw,
Pas encore testé mais a priori aucun breaking change d'un point de vue ansible entre les 2. La sécurité est déjà déployée et testée via ce role lorsque l'on dispose d'une licence. Surcharger les variables es_version, es_api_basic_auth_username et es_api_basic_auth_password devrait suffire. Dès que la 7.0.0 sera releasée sur Ansible Galaxy (#573), je verrais pour releaser un 7.1.0.


Hello @wixaw,
Elasticsearch 7.1 is not tested yet with this Ansible role, but there is no breaking changes from ansible view. Security is already deployed and tested using this role when you provide it some license. Overriding variables es_version, es_api_basic_auth_username and es_api_basic_auth_password should be enough. As soon as 7.0.0 is released on Ansible Galaxy (#573), We'll work on 7.1.0 release.

@wixaw
Copy link

wixaw commented Jun 4, 2019

OK, thanks
I'm rewriting the playbook, I keep you informed about the smooth running

@jmlrt jmlrt closed this as completed in #573 Jun 4, 2019
@jmlrt
Copy link
Member

jmlrt commented Jun 4, 2019

Hi everyone,
We finally decided do skip 7.0.0 release and directly release ansible-elasticsearch 7.1.1.
#573 was just merged.
The release is now available in GitHub and Ansible Galaxy.

@wixaw
Copy link

wixaw commented Jun 4, 2019

Yeahhh ! Thanks 👍

@abraxxa
Copy link
Author

abraxxa commented Jun 4, 2019

Thanks!

@mponton
Copy link

mponton commented Jun 4, 2019

@jmlrt Thanks and congrats on the release!

@fedelemantuano
Copy link

I installed a cluster with this role and I didn't change the default.

Can I move nodes folder from /var/lib/elasticsearch/host-host in /var/lib/elasticsearch?
Or for me is it better to change default (es_data_dirs: /var/lib/elasticsearch/{{ inventory_hostname }}-{{ inventory_hostname }})?

Multi-instances support was implemented when it was the only solution available to optimize usage of big servers with a lot of memory as well as allow cluster testing locally.

It came with a lot of drawbacks as it required to override Elastic official packages behavior by changing standard configuration path, data path and log path (even in single instance configuration) in this ansible role as in the puppet module we also provide.

One major point is the need to provide customized init scripts §(systemd or init depending on OS) instead of the ones provided by official packages.

This is bringing confusion to a lot of users and is responsible of a lot of the complexity and maintenance burden in our code.

Currently you are no more restricted to this only solution when needed to setup more than one instance of elasticsearch on the same host as Elastic is fully supporting Docker containers and is providing official Docker images.

We are now recommending Docker containers when there is some use-case which requires installing multiple elasticsearch instances in the same host.

That's why we are taking profit of Elasticsearch 7.0 major release to integrate this breaking change and remove multi-instances support.

This change should still allow you to change directory paths by overriding ansible variables as well as http and transport ports.

For existing multi-instances uses cases, the master branch is already working with elasticsearch 7.X and multi-instance support. You only need to use ansible-galaxy install git+https://github.com/elastic/ansible-elasticsearch.git,86c8e3c59bf2e34b62091cc766dd5f84df5fbef0 instead of ansible-galaxy install elastic.elasticsearch,7.0.1.

@jmlrt
Copy link
Member

jmlrt commented Jun 14, 2019

Hello @fedelemantuano,

We didn't document it and test it but moving some folders is possible of course.

Something like this should work:

  • stop elasticsearch before
  • move es folders:
mv /etc/elasticsearch/{{ instance_name }}/* /etc/elasticsearch/ && rm -fr /etc/elasticsearch/{{ instance_name }}
mv /var/lib/elasticsearch/{{ node_name }}-{{ instance_name }}/* /var/lib/elasticsearch/ && rm -fr /var/lib/elasticsearch/{{ node_name }}-{{ instance_name }}
mv /var/log/elasticsearch/{{ node_name }}-{{ instance_name }}/* /var/log/elasticsearch/ && rm -fr /var/log/elasticsearch/{{ node_name }}-{{ instance_name }}
mv /var/run/elasticsearch/{{ node_name }}-{{ instance_name }}/* /var/run/elasticsearch/ && rm -fr /var/run/elasticsearch/{{ node_name }}-{{ instance_name }}
  • update ansible role to 7.1.1 release: ansible-galaxy install elastic.elasticsearch,7.1.1
  • check that es_conf_dir, es_data_dirs, es_log_dir, es_pid_dir and es_instance_name aren't overridded in your playbook (remove them if needed)
  • run your playbook
  • check that elasticsearch is started or start it if needed

⚠️ Of courses as this hasn't been tested, please backup your datas and conf before and validate the procedure in a test environment. After the migration, validate again that your data are still here.

If your prefer waiting more tests, feel free to create an issue in the repo to request testing and documenting this procedure and we'll do it when we'll have a little more time to dedicate to ansible-elasticsearch role.

@fedelemantuano
Copy link

Hi @jmlrt,

thanks for your answer. I will open the issue because this can be a common problem.
I tested on test cluster but without move anything. The configuration are recreated, but I couldn't test the data because I understood this problem in test.

@zez3
Copy link

zez3 commented Mar 8, 2021

In light with the recent availability of dual ADM Epyc Rome CPU(128 Cores) Server architectures with NVMe disk and the trend of ever increasing the number of cores on a single Server, I would say that the multi-instances uses cases should be officially brought back from the dead.

I would also argue that K8s, docker or any other form of container orchestration would increase the overhead(by a minimum 3-5%) and latency compared to a bare metal installation.

Please do let me know your opinions on this?

@jmlrt

@jmlrt
Copy link
Member

jmlrt commented May 14, 2021

Hi @zez3,
Docker is still the recommended solution and adding back multi-instances isn't in the roadmap.

As mentionned in #554 (comment) multi-instance requires too much workarounds like rewriting init scripts and configuration files instead of using the ones provided in the official packages and ensuring they are still compliant with each new version.

This is not something we can afford too maintain and support as the setup using multi-instances with all these workaround may derive too much of the official packages.

Docker overhead is really minimal and I think the advantages outweigh the disadvantages here.

@zez3
Copy link

zez3 commented May 15, 2021

Hey Julien,
I see your point and position but the multiple init scripts seems nothing complex to me and at this time I do not see where the problem lies. We have Template Unit files(@) being used to create multiple instances of units in systemd working for some good years.
I'll probably have to hit my head on the described new version compliance through upcoming updates issues to come to your conclusion. I'll gladly do that by trying the multi-instance as a role on-top of the official role provided here to keep close with future updates and I will let you know.
I have to add my own cgroup limits, cpu/mem affinity in relation to my numa node architecture and nvme disks, anyway.

Regarding the last sentence I would argue that in our case the docker/container advantages will tip the balance in their favor. We don't play much with the productive cluster. Perhaps once a year or on occasion if there are some new "features"/bug fixes that we really need.
We'll also use the bundled java version to stay compliant.
Most updates will probably be on the new SIEM side(aka kibana).

I'm not sure what minimal means to you(please feel free to elaborate and corroborate some internal performance testing if available) but to me spending 3-5% at a minimum and >10% at a max from 128 cores results in >12 cores eaten; I could really use them somewhere else. That is just the CPU.

Regarding disk IOPS, latency and bandwidth, the overlayfs is another OVERLAY which is a virtual uper level filesystem over the real FS with his respective needed syncs.
From kernel 5.10 onward we have the new "volatile" sync option with the compromise of security. But which LTS Linux Distro runs such kernel version... We cannot run Experimental ones in production.
For those running containers on 4.16 kernel to v4.8 be aware:
https://www.cloudfoundry.org/blog/an-overlayfs-journey-with-the-garden-team/

Please don't get me wrong, I have nothing against containers and I fully agree that they have their usage. I use them myself where needed.
Like I said in our situation this overhead and advantages provided by containers do not make sense.
If interested in deep performance analysis please have a read:
here http://www.cs.binghamton.edu/~huilu/pubs/SOCC20.pdf
here https://doi.org/10.1109/SMARTCOMP.2018.00019
and here http://www-scf.usc.edu/~qiumin/pubs/MSST2017_camera_ready.pdf

@zez3
Copy link

zez3 commented May 16, 2021

And then I just found this:
elastic/elasticsearch#24867

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
7 participants