Other PID file issues

[alex-harvey-z3q]

I have just set up Kibana 4.3.1 from archive.

Aside from having to specify pid.file in $config in order to have the PID file line in the init.d script, I have another issue.

It seems to me that Kibana itself is trying to create its own PID file, and if Puppet creates it, there's a warning sent to Kibana's log file:

# cat /var/log/kibana4.log | jq .
...
{
  "message": "pid file already exists at /var/run/kibana.pid",
  "path": "/var/run/kibana.pid",
  "pid": "31244",
  "tags": [
    "pid",
    "warning"
  ],
  "@timestamp": "2016-01-22T07:48:44+00:00",
  "type": "log"
}

Unfortunately, Kibana doesn't then delete its PID file.

I'll submit a PR to show how I fixed it.

[alex-harvey-z3q]

PR raised #43

[lesaux]

Thanks for the PR @alexharv074.

Now that I look at it I can't remember while I was creating the pid file in the first place...

[alex-harvey-z3q]

Hi @lesaux I wasn't expecting you to merge it so quickly. If you're happy with this PR, then we should remove the other references to pid.file as well. I will raise another PR for this. However, I had assumed the issue with pid.file related to earlier versions although I haven't checked.

[lesaux]

I think the only references to pid.file that remain are in the init template. I think all is left is to edit the readme.

[alex-harvey-z3q]

Actually it's not going to be as easy as I thought. After blowing away the whole VM and reprovisioning I'm now having the same issue as a user of the evenup/kibana module is having jlambert121/jlambert121-kibana#65 I'll get back to you when I have fix.

[alex-harvey-z3q]

This is a real headache.

It seems that Kibana4 does try to manage its PID file, which defaults to /var/run/kibana.pid unless set in pid.file. However, the Kibana user needs read/write access to the PID file, which it won't have in the default location of /var/run unless Kibana is run as root.

In that case we'll see the following error in the log file when Kibana starts:

# cat /var/log/kibana4.log | jq .
...
{
  "error": {
    "code": "EACCES",
    "stack": "Error: EACCES, open '/var/run/kibana.pid'\n    at Error (native)",
    "name": "Error",
    "message": "EACCES, open '/var/run/kibana.pid'"
  },
  "message": "EACCES, open '/var/run/kibana.pid'",
  "level": "fatal",
  "pid": 8675,
  "tags": [
    "fatal"
  ],
  "@timestamp": "2016-01-23T08:59:31+00:00",
  "type": "log"
}

That's the Kibana app failing to create the PID file due to insufficient perms.

So I manually make a directory /var/run/kibana and set its perms:

[root@centos-66-x64 ~]# mkdir -p /var/run/kibana
[root@centos-66-x64 ~]# chown kibana:kibana /var/run/kibana/

Then I set pid.file = /var/run/kibana/kibana.pid and I alter your init template in Puppet to set pidfile to the same then puppet apply and great it's working. I tested stop and force-stop, and they work too.

If we go down this path, then we really need to have Puppet always automatically configure pid.file.

The alternative then is to have the init script deal with the PID file, using echo $! > $pidfile as you had originally.

In either case, the app doesn't delete the PID file again so the init script must do this.

So there's no simple solution here, and I have no idea whether earlier versions of Kibana behaved in the same way.

My preferred solution would be to make all of this as visible as possible to the user by having Puppet deal with it all:

• Puppet should firstly create /var/run/kibana with correct perms
• Remove the feature allowing the user to choose their own PID file location. It's complicated enough as it is!
• Merge pid.file into $config silently so that the user no longer has control over this.

I'll send a PR shortly because I need to get this code working. If you would prefer another approach let me know and I'm happy to fix it up.

[alex-harvey-z3q]

To further complicate this the pid.file feature was introduced in 4.2. Before that the setting was pid_file.

[lesaux]

about pid.file vs pid_file: yes that's why I am looking for both variables in the init template (https://github.com/lesaux/puppet-kibana4/blob/master/templates/kibana.init.erb#L29), and why I decided to use a hash for the kibana.yml configuration when v4.2 was released, because that release broke all of my previous parameters.

I haven't had time to test things out myself recently, but I think I'll take a look today.

[alex-harvey-z3q]

I raised another PR #44 I think this is probably the best solution, caveat being that if a user leaves pid.file on the default setting and tries to run as a non-root user, they'll hit the same issue I did. Documentation would need to be updated to warn people about this, and possibly I could add some sanity checking too.

[lesaux]

I'm not sure what's the status of elastic/kibana#4680, but it seems the creation of the pid file is dependent on the fact that kibana can connect to elasticsearch or not.

In my vagrant file to manually test the module, I wasn't starting up an elasticsearch service, which probably explains why I wasn't seeing the problem.

If that issue is accurate, we shouldn't be even creating the pid file at all, as kibana should be creating it.

[alex-harvey-z3q]

Right. The PR I raised that you merged already stopped the module from creating the PID file.