Elasticsearch 7.4.0 does not start on ECK 1.0.0-beta1

[kuisathaverat]

Bug Report

What did you do?

We were trying to deploy Elasticsearch 7.4.0

What did you expect to see?

Five Elasticsearch nodes up and running.

What did you see instead? Under which circumstances?

Five unhealthy Elasticsearch nodes, after reading the logs there is an error relates to max virtual memory areas

Environment

• ECK version:

  1.0.0-beta1

• Kubernetes information:

  insert any information about your Kubernetes environment that could help us:

  • Cloud: GKE

  for each of them please give us the version you are using

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-19T13:57:45Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-gke.24", GitCommit:"2ce02ef1754a457ba464ab87dba9090d90cf0468", GitTreeState:"clean", BuildDate:"2019-08-12T22:05:28Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}

• Resource definition:

---
apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
kind: Elasticsearch
metadata:
  name: elastic-stack
spec:
  version: 7.4.0
  nodeSets:
  - name: default
    config:
      node.master: true
      node.data: true
      node.ingest: true
    count: 5

The error in logs is

ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

I have tried to configure the node as privileged without any luck

---
apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
kind: Elasticsearch
metadata:
  name: elastic-stack
spec:
  version: 7.4.0
  nodeSets:
  - name: default
    config:
      node.master: true
      node.data: true
      node.ingest: true
    podTemplate:
      spec:
        setVmMaxMapCount: true
        containers:
        - name: elasticsearch
          securityContext:
            privileged: true
    count: 5

so finally I've tried a sidecar container to change the settings and it works, but it is weird if you are using the default image.

---
apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
kind: Elasticsearch
metadata:
  name: elastic-stack
spec:
  version: 7.4.0
  nodeSets:
  - name: default
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.ml: true
    podTemplate:
      spec:
        setVmMaxMapCount: true
        containers:
        - name: init-sysctl
          image: busybox
          command: ["sh", "-c", "sysctl -w vm.max_map_count=262144 && sleep 2147483647"]
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
    count: 5

[pebrc]

https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-virtual-memory.html describes the init container approach you used. Alternatively you can disable memory mapped files (e.g. for non-production environments node.store.allow_mmfs: false Or you can set these these kernel settings with a daemon set ahead of time for all nodes.

[sebgl]

Just to make it clear: we removed the setVmMaxMapCount option from the Elasticsearch spec starting ECK 1.0.0-beta1.

Not an easy decision, see the corresponding issue. The main reason is that it required Pods to be privileged, which is not what most users would expect an Elasticsearch operator would do. Power is now in the hands of the users, as @pebrc describes.