procstat response when monitored process dies

[aodj]

Could we improve the response of the procstat input such that when a process dies, it still continues to send some metrics, ideally something signalling that the process has died. Currently, the input outputs a message saying Error: procstat getting process, exe: [] pidfile: [] pattern: [hekas] Failed to execute pgrep. Error: 'exit status 1' (or similar when using pid_file).

[sparrc]

@aodj What would you suggest that procstat sends when the process dies?

[aodj]

I'm not sure exactly. I initially thought that you could send a process_alive measurement, which changes to false when the /proc endpoint returns file not existing, but then you're sending an extra value with each metric push. It could be an opt in setting like an include_heart_beat option in the input, but I was basically looking for some way to determine when a process has died, without having to rely upon the time since last metric received.

[sparrc]

opting-in to that seems fine

[asangal]

This issue may happen if you have multiple entries in one file:

[[inputs.procstat]]
     exe = "telegraf"
     prefix = "pgrep_svcs_process"

[[inputs.procstat]]
     exe = "jenkins"
     prefix = "pgrep_svcs_process"

[[inputs.procstat]]
     exe = "docker"
     prefix = "pgrep_svcs_process"

To fix it, you can just use a pattern "." or "telegraf|." for exe = <exe_pattern> and then you don't have to create N no. of separte files.

[[inputs.procstat]]
    exe = "telegraf|.*"
    prefix = "pgrep_svcs_process"

I preferred "telegraf|." over just ".*" or just "." as it'll make it more easy to tell who's doing the pattern matching if you use exe = <..> in a query.

[fchiorascu]

Hi,

Issue:
2017-12-12T14:33:36Z E! Error in plugin [inputs.procstat]: E! Error: procstat getting process, exe: [memcached] pidfile: [] pattern: [] user: [] Error running /bin/pgrep: exit status 1
2017-12-12T14:33:37Z E! Error in plugin [inputs.procstat]: E! Error: procstat getting process, exe: [java] pidfile: [] pattern: [] user: [] Error running /bin/pgrep: exit status 1
2017-12-12T14:33:37Z E! Error in plugin [inputs.procstat]: E! Error: procstat getting process, exe: [nginx] pidfile: [] pattern: [] user: [] Error running /bin/pgrep: exit status 1

What I want to achieve:
-To monitor all processes and to have the metrics available in Grafana.

• Unfortunately I've tried many approaches proposed on internet and I don't have Graphs in Grafana.

Recommendation how to look the procstat config?
[[inputs.procstat]]
exe = "?"
prefix = "?"

Kind Regards,

[danielnelson]

@fchiorascu You should be able to do this:

[[inputs.procstat]]
  exe = "memcached"
  prefix = "memcached"

Unfortunately, the procstat plugin does currently show this error when the process is not running, so make when you run pgrep memcached it matches a process.

[danielnelson]

I wrote a short proposal on how I think we should fix this in #4237

[danielnelson]

The solution in #4237 has been implemented for 1.8.