[Filebeat stable/filebeat] Modules Don't Enable

[chenchik]

Describe the bug
When trying to use the filebeat modules, they aren't enabled.

Version of Helm and Kubernetes:
Helm
Client: &version.Version{SemVer:"v2.7.2", GitCommit:"8478fb4fc723885b155c924d1c8c410b7a9444e6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}

Kubernetes
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-gke.19", GitCommit:"bebe882824db5431820e3d59851c8fb52cb41675", GitTreeState:"clean", BuildDate:"2019-07-26T00:09:47Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
stable/elastic-stack
stable/filebeat:7.0.1

What happened:
Cannot enable modules unless manually exec'ed into the pod.

What you expected to happen:
To be able to enable modules using helm values.

How to reproduce it (as minimally and precisely as possible):
Include the following in your values.yaml file

overrideConfig:
  filebeat.config:
    modules:
      enabled: true
      path: ${path.config}/modules.d/*.yml
      reload.enabled: true

Anything else we need to know:
I have tried to set this underneath config instead of overrideConfig (example below)

  config:
    filebeat.config:
      modules:
        enabled: true
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: true

Which resulted in the following behavior when exec'ed into the pod

sh-4.2# filebeat modules list
Error in modules manager: modules management requires 'filebeat.config.modules.path' setting

[eyespies]

I was just investigating the use of modules and found that in the source Docker image the modules are all named .yml.disabled:

[ [email protected] / champgrid ] 09:17 $ docker run -it docker.elastic.co/beats/filebeat-oss:7.0.1 /bin/bash
bash-4.2$ cd modules.d
bash-4.2$ pwd
/usr/share/filebeat/modules.d
bash-4.2$ ls
apache.yml.disabled         haproxy.yml.disabled  kafka.yml.disabled     mongodb.yml.disabled  osquery.yml.disabled     santa.yml.disabled
auditd.yml.disabled         icinga.yml.disabled   kibana.yml.disabled    mysql.yml.disabled    postgresql.yml.disabled  system.yml.disabled
elasticsearch.yml.disabled  iis.yml.disabled      logstash.yml.disabled  nginx.yml.disabled    redis.yml.disabled       traefik.yml.disabled

There is one workaround I am sure will work (but haven't tried it) and one that I think will work (but also have not tried):

1. create a custom container with the modules you want enabled (99% sure this will work)
2. specify a custom command via .Values.command to rename any .disabled files to be not disabled

I am not sure if #2 is the official, correct, way of doing this, but it would be nice to get an option in the values.yaml to list the modules to be enabled and then the startup process takes care of the rename (if it exists I couldn't find it). The process would have to account for someone typing a new module file that doesn't exist (or one that is a typo) which I think may best be be handled by allowing the container to start and just emitting a message that the module file was not found.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.