changes in kibana package

[lesaux]

New kibana packages (4.4.0-1) now provide:

• the creation of a kibana user and group (999:999)
• an init file /etc/init.d/kibana
• the init file creates the log directory /var/log/kibana
• the init file creates a pid file /var/run/kibana.pid (which doesn't seem consistent with pid.file option in the server parameters).

pid.file:

Specifies the path where Kibana creates the process ID file.

@jordansissel has the following comment in the init script

  # process by this pid is running.
  # It may not be our pid, but that's what you get with just pidfiles.
  # TODO(sissel): Check if this process seems to be the same as the one we
  # expect. It'd be nice to use flock here, but flock uses fork, not exec,
  # so it makes it quite awkward to use in this case.

We need to take into account all of these changes for the "package" installation method.

[lesaux]

So it seems the two installation methods (archive vs package) will now have different purposes. The "package" method will provide a more rigid approach, while the archive installation method will keep the flexibility we've had until now.

I think I will overhaul all of the module parameters, to make clearer the separation between the "archive" and "package" installation methods. Parameters relevant to each install method would be prepended by either archive_ or package_.

Here's what I have in mind:

# == Class: Kibana4
#
# Installs and configures Kibana4.
#
# === Parameters
#
# Parameters that start with "package" are relevant to the "package" installation method,
# while parameters that start with "archive" are relevant to the "archive" installation method.
#
# [*version*]
# Version of Kibana4 that gets installed.  Defaults to the latest 4.4.0 version
# available at the time of module release.
#
# [*install_method*]
# Set to 'archive' to download Kibana from the Elasticsearch download site (see
# also `package_download_url` below).  Set to 'package' to use the default package
# manager for installation.  Defaults to 'archive'.
#
# [*package_name*]
# The name of the Kibana4 package that gets installed. Defaults to 'kibana'.
#
# [*package_use_official_repo*]
# Use official apt or yum repository. Only used if package_provider is set to 'package'.
#
# [*package_repo_version*]
# apt or yum repository version. Only used if 'use_official_repo' is set to 'true'.
# defaults to '4.4'. We should evaluate this automatically.
#
# [*archive_download_url*]
# Alternative URL from which to download Kibana iff `package_provider` is
# 'archive'. Defaults to `undef`, because by default the URL is constructed
# from the usual Elasticsearch download site URL, the `package_name` and
# `package_ensure`.
#
# [*archive_proxy_server*]
# Specifies which proxy server use to download archive. Valid format is
# http[s]//[user:passwd@]proxy_host:port. Not supported when `archive_provider`
# is 'nanliu' or 'puppet'.
#
# [*archive_provider*]
# Select which `archive` type should be used to download Kibana from the
# Elasticsearch download site. There exist at least two modules that provide an
# `archive` type: "camptocamp/archive" and "nanliu/archive" (or "puppet/archive"
# since the module is now in the care of puppet-community). Defaults to
# 'camptocamp'. If you set this to 'nanliu' (or 'puppet') make sure you have that
# module installed since both cannot be recorded as a dependency in metadata.json
# at the same time.
#
# [*archive_manage_init_file*]
# Install init file. If the init script is provided by a package,
# set it to `false`. Defaults to 'true'
#
# [*archive_init_template*]
# Service file as a template
#
# [*archive_manage_user*]
# Should the user that will run the Kibana service be created and managed by
# Puppet? Defaults to 'false'.
#
# [*archive_kibana4_group*]
# The primary group of the kibana user
#
# [*archive_kibana4_gid*]
# The group ID assigned to the group specified in `kibana4_group`. Defaults to `undef`.
#
# [*archive_kibana4_user*]
# The user that will run the service. For now installation directory is still owned by root.
#
# [*archive_kibana4_uid*]
# The user ID assigned to the user specified in `kibana4_user`. Defaults to `undef`.
#
# [*archive_install_dir*]
# Installation directory used iff install_method is 'archive'
# Defaults to '/opt'.
#
# [*archive_symlink*]
# Determines if a symlink should be created in the installation directory for
# the extracted archive. Only used if install_method is 'archive'.
# Defaults to 'true'.
#
# [*archive_symlink_name*]
# Sets the name to be used for the symlink. The default is '$install_dir/kibana4'.
# Only used if `package_provider` is 'archive'.
#
# [*archive_service_name*]
# Name of the Kibana4 service. Defaults to 'kibana4'.
#
# [*service_ensure*]
# Specifies the service state. Valid values are stopped (false) and running
# (true). Defaults to 'running'.
#
# [*service_enable*]
# Should the service be enabled on boot. Valid values are true, false, and
# manual. Defaults to 'true'.
#
# [*config_file*]
# The location, as a path, of the Kibana configuration file.
#
# [*babel_cache_path*]
# Kibana uses babel (https://www.npmjs.com/package/babel) which writes it's cache to this location
#
# [*config*]
# A hash of key and values that describe the kibana server parameters (kibana.yml)
#

Let me know what you think.

[alex-harvey-z3q]

I think a design goal ought to be to maintain a level of design consistency in this module with that of the Puppet modules released by Elastic.co. After all, most people will be using Elastic.co's modules for Elasticsearch & Logstash and this one for Kibana.

Also, I would assume that Elastic.co will eventually get their act together and release RPMs for all versions of Kibana 4 so that we could hope to one day drop support for archive installation.

Finally, I don't see any reason why a package installation should be less flexible than an archive installation. For instance, I want to control the Kibana's user and group IDs but I also want to install the RPM. Therefore, I want the module to allow me to set the UID and have the module ensure that User['kibana'] -> Package['kibana4'].

[alex-harvey-z3q]

The other point I'd make is it sounds like they need to clean up their code before you try to support their package.

[egouraud]

great exactly what i was looking for :)

here are my comments:

• I think it should more convenient to use more "elastic-like" variable names (manage_repo and repo_version instead of package_use_official_repo and package_repo_version) for compliance purpose.
• agree with @alexharv074 concerning RPM package

I'm really looking after this release ;) thanks for your job

cheers,

[lesaux]

Hi,

I really wanted to make the parameter names consistent between the two installation methods. That's why I though it would be clearer to prepend all parameters relevant to the "package" installation by the "package_" prefix, and all parameters relevant to the "archive" installation method by "archive_". Any other parameters would be common to both methods.

package_repo_version should probably disappear and be evaluated anyways.

I understand the desire to keep the "package" installation method as flexible as the "archive" method, however, because of all the recent additions to the packages (creation of a kibana user, addition of init script) this would become more complex and more difficult to keep track of. Depending on the scenario, we'd have to undo things that the package installation performed. For example we'd have to be able to remove the user completely if you had your setup integrated with ldap, or remove the init script if you wanted to use upstart or systemd. The pid.file param is also a big mess.

It might be simpler to start of fresh by limiting the "package" installation method features, and progressively adding some of those features back.

I'm expecting to have some time to work on the module again in the upcoming days.

[alex-harvey-z3q]

It's not complex at all to give the same functionality and flexibility to both the archive and package installation. You just need to make sure the ordering is correct:

group { 'kibana4':
  ensure => present,
  gid => $kibana4_gid,
}
user { 'kibana4':
  ensure => present,
  uid => $kibana4_uid,
  gid => $kibana4_gid,
}
file { '/etc/init.d/kibana':
  ensure => file,
  content => ...,
}
User['kibana4'] -> Package['kibana'] -> File['/etc/init.d/kibana']

There's no need to undo anything.

[lesaux]

Thank you for the lesson on ordering puppet resources...

You're missing some of the underlying issues. What if I don't want my user to be named "kibana4", or if I would like my init script to be named something else than "kibana" (both of which are currently possible). The module would need to remove the user and init file created by the package installation in this case.

[alex-harvey-z3q]

Sorry, I didn't intend a lesson in ordering.

What I can say is if a package installation is supported by Elastic.co then no one will ever use the archive installation. We should obey the "keep-it-simple" principle, and encourage our users to also follow this principle.

If people want to do crazy, complex stuff, I wouldn't support that. If the module has features that the Elastic.co modules don't have, I would deprecate those features now. I don't see how anyone can use this module in support of LDAP user integration if the Elastic.co modules don't support this. And I have no idea why anyone would put their Kibana user in LDAP in the first place.

But the main point is, if you're going to break backwards compatibility, I hope you'll do so reluctantly and with very good reason.

[egouraud]

Hi there,

OK for the parameter name consistency, i understand you want to have a comprehensible way to use your module and it's fine.

I think I have answered too fast concerning package in my last message, obviously a package installation is always less flexible than an archive installation method. As you said, it will fix some things like users, working path, etc... which restrict post configuration possibilities. @alexharv074 has explained an argument about controlling user's UIG/GID, which does not implies to change what a package have done but only to do some things before the package installation and i agree with that :)

It might be simpler to start of fresh by limiting the "package" installation method features, and progressively adding some of those features back.

That's exactly what you need, let the package do his job and if needed fix what the package maintainer has forgotten :)

cheers,

[alex-harvey-z3q]

@lesaux Let me add a big thank you for the work to write and maintain this module. It was very frustrating for me that I had to use a tar ball archive installation in production, and that's totally on Elastic.co, but all the work you put into this module did make it very easy.

[lesaux]

Hey @alexharv074, sorry I was a bit snappy yesterday - bad day at work.

I'm currently setting up my dev and testing environments to resume work on the module.

[alex-harvey-z3q]

👍

[nickchappell]

Any updates on this?

[lesaux]

I'll close this as I've released a new module version