Setting up modules when Logstash is running as a service

[ycombinator]

Currently, in order to setup Logstash modules (e.g. netflow), there is a first-time setup command that must be run, like so:

bin/logstash --modules MODULE_NAME --setup [-M "CONFIG_SETTING=VALUE"]

This command creates the necessary artifacts in Elasticsearch and Kibana such as Elasticsearch templates, Kibana visualizations, and Kibana dashboards. It also starts up a Logstash pipeline for the given module, which keeps the Logstash process running.

Thereafter, to create additional pipelines for the given module, the same command sans --setup must be invoked:

bin/logstash --modules MODULE_NAME [-M "CONFIG_SETTING=VALUE"]

This pattern works well when Logstash is installed as a standalone program, typically from its .tar.gz or .zip build artifact.

However, when Logstash is installed as a service, typically from its .deb or .rpm build artifact, this pattern starts to break down. Specifically, with Logstash running as a service, how can the first-time --setup step be invoked?

[ycombinator]

FWIW, this (specifically step 4 below) is how Beats seems to be solving for this use case, by making the Beat in question a service and a command both:

[ycombinator]

For the Beats pattern to be applied to Logstash, two changes would be required:

1. Make logstash available as a command as well as a service. This would require placing the logstash executable in the system path. Currently Logstash is installed as a service but can be executed as a command if the install path is known (e.g. when installed via a .deb package, Logstash can be executed as a command by running /usr/share/logstash/bin/logstash).

2. Make logstash --modules MODULE_NAME --setup run and exit. Currently the command starts up the pipeline for the module so continues to run without exiting.

[ycombinator]

These are the minimal changes required to deliver an MVP:

In 6.0.0 and 6.1.0

• Honor --setup as-is (continues running the process)

In 6.2.0 - 6.x

• Honor --setup as-is (continues running the process)
• Also provide --setup-and-exit flag which will do the setup and then exit (Add new --setup-and-exit flag for Logstash module setup #8767)
• If both --setup and --setup-and-exit are specified, fail and let user know the flags are mutually exclusive.
• Add --setup-and-exit to docs

In 7.0.0

• Change the meaning of --setup to do the setup and then exit
• Deprecate --setup-and-exit (make it do the same thing as before but also log a warning about using --setup instead and this flag becoming obsolete in the future)
• Update docs to use --setup

In 8.0.0

• Obsolete --setup-and-exit (fail with error message to use --setup instead)

[acchen97]

@ycombinator as discussed yesterday, I think we should add the --setup-and-exit option and still retain the --setup command for the long term as there are use cases for both.

For deb/rpm running as a service:

1. Run --setup-and-exit to bootstrap
2. Run module as a service without additional setup options

For tar/zip standalone: just run --setup to both bootstrap and spin up LS pipeline

/cc @tbragin

[ycombinator]

I'm good with this approach. @jordansissel what are you thoughts, since it differs from our original proposal?

[markandrewj]

I came across this today, a --setup-and-exit flag would be useful.