Lots of errors with influxdb multiple fields and graphite protocol

[shuaiming]

description

I've got a lot of errors in my log file, hundred megabytes per day.
How can I reduce the log size, or I missed anything in the document.

[load_midterm:alert2] 2016/03/29 16:47:13 E! error evaluating expression: name "longterm" is undefined. Names in scope: shortterm,group,host,plugin,time
[load_midterm:alert2] 2016/03/29 16:47:13 E! error evaluating expression: name "longterm" is undefined. Names in scope: midterm,group,host,plugin,time

analysis

collectd ----graphite protocol----> influxdb ----subscriptions----> kapacitor

1. collectd write to influxdb by graphite protocol and tcp

2. influxdb has graphite plugin:

[[graphite]]
  enabled = true
  database = "graphite"
  bind-address = ":2003"
  protocol = "tcp"
  separator = "_"
  templates = [
     ...
     "*.*.*.load.* .group.host.plugin.measurement.field"
     ...
  ]

3. kapacitor task "load_midterm" is:

Name: load_midterm
Error:
Type: stream
Enabled: true
Executing: true
Databases Retention Policies: ["graphite"."default"]
TICKscript:
var load_longterm = stream
  .from()
  .measurement('load')
  .groupBy('host')


load_longterm.alert()
  .id('kapacitor/{{ .Name }}')
  .message('{{ .Level }} {{ index .Tags "host" }} {{ .ID }} is value:{{ index .Fields "longterm" }}')
  .crit(lambda: "longterm" > 3)
  .log('/var/log/kapacitor/alerts.log')


DOT:
digraph load_midterm {
graph [throughput="2238.88 points/s"];

srcstream0 [avg_exec_time_ns="0" ];
srcstream0 -> stream1 [processed="320715369"];

stream1 [avg_exec_time_ns="39.534µs" ];
stream1 -> alert2 [processed="2547921"];

alert2 [alerts_triggered="31253" avg_exec_time_ns="706.372µs" ];
}

4. TCP Stream of graphite piping to influxdb from tcpdump:

wheezy.xxxxxx-wcl.xx-hadoop-2.load.load.shortterm 0.000000 1459240797
wheezy.xxxxxx-wcl.xx-hadoop-2.load.load.midterm 0.020000 1459240797
wheezy.xxxxxx-wcl.xx-hadoop-2.load.load.longterm 0.050000 1459240797

5. UDP Stream of influxdb piping to kapacitor from tcpdump:

load,group=xxxxxx-wcl,host=xx-hadoop-2,plugin=load shortterm=0 1459240797000000000
load,group=xxxxxx-wcl,host=xx-hadoop-2,plugin=load midterm=0.02 1459240797000000000
load,group=xxxxxx-wcl,host=xx-hadoop-2,plugin=load longterm=0.05 1459240797000000000

6. influxdb query is :

> select * from load where host='xx-hadoop-2' and time > 1459240797000000000 and time < 1459240827000000000;
name: load
----------
time            group       host        longterm    midterm plugin  shortterm
1459240807000000000 xxxxxx-wcl  xx-hadoop-2 0.05        0.02    load    0
1459240817000000000 xxxxxx-wcl  xx-hadoop-2 0.05        0.02    load    0

It seems influxdb must writing to kapacitor with all fields key in one line each time.

and sorry about my poor english.

[shuaiming]

the versions I've forgetton to say :
Kapacitor 0.11.0
InfluxDB version 0.10.3

[nathanielc]

@shuaiming I have added a .quiet property to the EvalNode so you can suppress errors if your data often has missing fields. This change is part of an upcoming bugfix PR.

[devfacet]

@nathanielc : can you show an example for using quiet()?

UPDATE:

I have something like;

stream
  |from()
    .measurement('docker_cpu')
    .groupBy('host', 'cont_image', 'cont_id')
  |eval(lambda: 0.0 + "usage_percent")
    .as('usage_p')
    .quiet()
  |alert()
    .id('{{ index .Tags "host" }}/{{ .Name }}/{{ index .Tags "cont_image" }}')
    .message('{{ .ID }} is {{ .Level }}: MEM% {{ index .Fields "usage_p" | printf "%0.2f" }}')
    .warn(lambda: "usage_p" > 95)

and it throws errors like;

[docker-cpu:alert3] 2016/04/09 02:57:08 E! error evaluating expression: name "usage_p" is undefined. Names in scope: time,host,cont_id,cont_image,cont_name,cpu
[docker-cpu:alert3] 2016/04/09 02:57:08 E! error evaluating expression: name "usage_p" is undefined. Names in scope: time,host,cont_id,cont_image,cont_name,cpu
[docker-cpu:alert3] 2016/04/09 02:57:08 E! error evaluating expression: name "usage_p" is undefined. Names in scope: cont_id,cont_image,cont_name,cpu,host,time