Add env vars to libbeat.yml default conf file

[marminthibaut]

Proposal to provide a default libbeat configuration with common environment variables defined.

What do you think about introducing these variables?

Thx!

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.

[ruflin]

Hi @marminthibaut. As I'm not too big of a fan of the environment variables, so I prefer not to have it in the default config as this would even encourage the usage. But that is my personal opinion, lets see what the rest of the team thinks.

[marminthibaut]

For Dockerbeat, we are generating a dockerbeat-docker.yml configuration file used by a Dockerfile to generate the dockerbeat docker image. With docker, environment variables provides a lot of flexibility to launch containers without injecting custom configuration files.

I see two solutions:

• include common env vars into the libbeat.yml configuration file
• develop an embedded way to generate custom libbeat configuration section specific for each beat (from makefile, scripts, ?)

Any idea?

[ruflin]

I would then choose the second option. Perhaps it makes sense the we make the path to libbeat.yml configurable, so running make update selects your own file. There is the issue that you have to keep it up-to-date, but not sure how else this could be solved without adding lots of complexity.

One note here: We are discussing to potentially shorten the default configuration and only have the most important options in, as the config file gets longer and longer. This would simplify your task of keeping the files in sync.

[marminthibaut]

And what about having a libbeat template configuration file, then default
values should be set in the scripts/Makefile, but these variables could be
overwritten by the specific beat Makefile?

On Tue, Apr 19, 2016, 1:57 PM Nicolas Ruflin [email protected]
wrote:

I would then choose the second option. Perhaps it makes sense the we make
the path to libbeat.yml configurable, so running make update selects your
own file. There is the issue that you have to keep it up-to-date, but not
sure how else this could be solved without adding lots of complexity.

One note here: We are discussing to potentially shorten the default
configuration and only have the most important options in, as the config
file gets longer and longer. This would simplify your task of keeping the
files in sync.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1419 (comment)

[ruflin]

The big question that brings that up, which option should we make configurable?

An alternative idea: What you is basically overwrite the defaults. An alternative would be that we allow to overwrite the defaults on a code base. That means the variables would still be commented out but use the defaults you defined in the code of dockerbeat. If not commented out anymore, it will use the ones in the config file. The exception here is hosts ...

[tsg]

@marminthibaut As @ruflin mentioned, we plan to reduce a bit the size of the default configurations: #1417

Since we'll keep just the very common options that are not likely to change too often, I think you don't have to use our libbeat.yml, but provide your own version, so you can add the env variables.

[urso]

config file handling is based on github.com/urso/ucfg. There are two features planed which might help in general:

• variable expansion (config file internal + environment variables): Variable expansion go-ucfg#14
• CLI flag support: CLI flags support go-ucfg#15

CLI flag support is about merging command line flags into parsed configs -> CLI flags taking precedence.

for example configuring the elasticsearch host can be done like this:

1. config file with default value:

output.elasticsearch.hosts: ["localhost:9200"]

and reconfigure on start up beat using ES_HOST environment variable:

$ ./filebeat --output.elasticsearch.hosts.0=${ES_HOST}:9200

alternatively (using variable support) this can be shortened to:

1. config file

es.host: localhost
output.elasticsearch.hosts: ["${es.host}:9200"]

and

1. CLI

$ ./filebeat --es.host=${ES_HOST}

I think this solution is a good compromise without having to resort on environment variable expansion in libbeat.yml by default.

We might even consider merging multiple config files overwriting variables (but we've no particular plans for this feature).

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.

[ruflin]

@marminthibaut Did you find a solution on your side here? @urso is progressing on elastic/go-ucfg#14 which will give some interesting options, but it is not yet in libbeat. I will close this PR for the moment as we probably will not have environment variables in our default config.

@marminthibaut Please feel free to reopen it if you think we should proceed here.