[Question][logparser] Best approach for logparser generating two metrics

[kir4h]

Hi,

I would like some guidance on the best approach for my scenario.

• Telegraf + Prometheus + Grafana
• I am using logparser for parsing multiple log files on a folder. All of them look for a pattern which contains the timestamp and a numeric value (duration)
• I am interested in getting the duration value, but also the timestamp, since I will be using each of them in dashboards.

By setting a pattern, I can define one of them as the measurement and the other as a label, but I don´t seem to be able to issue two different metrics.

Is there a way two accomplish this with the logparser input plugin? Or should I use a processor plugin instead? (maybe the clone would do?)

I guess I could just add two logparser blocks over the same log files, each of them issuing a metric, but I guess it´s not the best way to go?

Thanks in advance!

[danielnelson]

The general idea is to parse each value in the log as a field, not as the measurement/tags. It will be helpful to think in terms of the Telegraf/InfluxDB model even though you are outputting to prometheus, each log line will create a single Telegraf metric (We usually report durations and timestamps in nanoseconds):

my-resource,host=example.org duration=5000000000,timestamp=1586037432000000000

In the prometheus output, use metric_version = 2 and each field will become a prometheus metric:

duration{host="example.org"} 5000000000
timestamp{host="example.org"} 1586037432000000000

[kir4h]

Thanks for your answer @danielnelson! I was indeed more thinking in terms of Prometheus than InfluxDB, without considering that it's only when it comes to the output that everything gets transformed into Prometheus. I've been reading a little about InfluxDB now to get familiar about the terms.

I have followed the example in logparser but can't still get the timestamp metric.

  [[inputs.logparser]]
   files = ["/var/log/mycustomlog/**.log"]
    [inputs.logparser.grok]
      patterns = ['%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} total time execution: %{NUMBER:duration_seconds:int}']
      measurement = "script"

With my input being

2020-04-05 09:00:14 total time execution: 13 seconds

And my output config

[[outputs.prometheus_client]]
   ...
metric_version = 2
string_as_label = true
export_timestamp = true
   ...

With these settings, the metric generated is only script_duration_seconds. I would expect to have also script_timestamp, am I missing something? (maybe related to my prometheus output config)

$ curl -s localhost:9273/metrics | grep "script_"
# HELP script_duration_seconds Telegraf collected metric
# TYPE script_duration_seconds untyped
script_duration_seconds{host="adb",path="/var/log/mycustomlog/mylog.log"} 13

Also from your comment it would be better to parse duration as a type duration, but I guess I can't do that without the log being 13s instead of 13 seconds.

EDIT:

After some initial troubleshooting:

• %{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} makes the timestamp a measurement value, which is not the approach you suggested (everything as field)
• %{TIMESTAMP_ISO8601:timestamp} makes the timestamp a field, but since it's a string it becomes a label in prometheus output, so I need somehow to transform this to an epoch in between

[kir4h]

Realized what I actually needed is that the metric is issued with its related timestamp (this way the metric represents the actual timestamp and not the scrape one). I realized this was not issued, even when export_timestamp is set to true in prometheus output plugin and noticed the code was only introducing it for version 1 of the metrics.

I raised #7289 to fix the behavior (if this is the intended behavior I will drop the PR).

Closing this question