panic: runtime error: invalid memory address or nil pointer dereference when running a replay when tick script contains post()

[pshemk]

Hi,

I was following instructions on https://docs.influxdata.com/kapacitor/v1.3/introduction/getting_started/

Kapacitor is running in a container from the official image tagged :latest

Tick script i tried to use:

root@kapacitor:/var/lib/kapacitor# kapacitor show load_alert
ID: load_alert
Error:
Template:
Type: stream
Status: disabled
Executing: false
Created: 05 Sep 17 00:31 UTC
Modified: 05 Sep 17 00:31 UTC
LastEnabled: 01 Jan 01 00:00 UTC
Databases Retention Policies: ["telegraf"."autogen"]
TICKscript:
stream
    |from()
        .measurement('system')
        .where(lambda: "alerting_load" == 'over3')
    |alert()
        .crit(lambda: "load1" > 0.1)
        .post('http://nagios-api:8080/message')
        .log('/tmp/alerts.log')

DOT:
digraph load_alert {
stream0 -> from1;
from1 -> alert2;
}

I recorded the stream:

root@kapacitor:/var/lib/kapacitor# kapacitor record stream -task load_alert -duration 20s
e25d8724-82e6-4da9-be6e-6cf6ba797da2
root@kapacitor:/var/lib/kapacitor# rid=e25d8724-82e6-4da9-be6e-6cf6ba797da2
root@kapacitor:/var/lib/kapacitor# kapacitor list recordings $rid
ID                                   Type    Status    Size      Date
e25d8724-82e6-4da9-be6e-6cf6ba797da2 stream  finished  284 B     05 Sep 17 00:41 UTC

But when I tried to replay it:

root@kapacitor:/var/lib/kapacitor# kapacitor replay -recording $rid -task load_alert

container crashed:

[httpd] 127.0.0.1 - - [05/Sep/2017:00:42:19 +0000] "POST /kapacitor/v1/replays HTTP/1.1" 201 258 "-" "KapacitorClient" 1283059a-91d3-11e7-8079-000000000000 11429
[task_master:1e50f514-94da-482c-a39e-a96446a056a2] 2017/09/05 00:42:19 I! opened
[task_master:1e50f514-94da-482c-a39e-a96446a056a2] 2017/09/05 00:42:19 I! closed
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x66246f]

goroutine 153 [running]:
panic(0x1cd80c0, 0xc42000c060)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/influxdata/kapacitor.newAlertNode(0xc4201ad7a0, 0xc420554000, 0xc420789130, 0xc420827918, 0x528a03, 0x1e45cc0)
        /root/go/src/github.com/influxdata/kapacitor/alert.go:365 +0x24af
github.com/influxdata/kapacitor.(*ExecutingTask).createNode(0xc4201ad7a0, 0x2e423a0, 0xc420554000, 0xc420789130, 0xc420789130, 0xc420762e80, 0x14, 0x2)
        /root/go/src/github.com/influxdata/kapacitor/task.go:464 +0x4f7
github.com/influxdata/kapacitor.(*ExecutingTask).link.func1(0x2e423a0, 0xc420554000, 0x0, 0x0)
        /root/go/src/github.com/influxdata/kapacitor/task.go:164 +0x1c6
github.com/influxdata/kapacitor/pipeline.(*Pipeline).Walk(0xc4207786e0, 0xc420827a70, 0x90, 0x1efe9e0)
        /root/go/src/github.com/influxdata/kapacitor/pipeline/pipeline.go:142 +0x6e
github.com/influxdata/kapacitor.(*ExecutingTask).link(0xc4201ad7a0, 0xc4201ad7a0, 0xc420368630)
        /root/go/src/github.com/influxdata/kapacitor/task.go:180 +0x5b
github.com/influxdata/kapacitor.NewExecutingTask(0xc42034f680, 0xc420793ec0, 0x2, 0x2, 0xc4206e5180)
        /root/go/src/github.com/influxdata/kapacitor/task.go:126 +0x217
github.com/influxdata/kapacitor.(*TaskMaster).StartTask(0xc42034f680, 0xc420793ec0, 0x0, 0x0, 0x0)
        /root/go/src/github.com/influxdata/kapacitor/task_master.go:422 +0x1f1
github.com/influxdata/kapacitor/services/replay.(*Service).doReplay(0xc4201f08f0, 0xc4207603c0, 0x24, 0xc420793ec0, 0xc42077f360, 0x0, 0x0)
        /root/go/src/github.com/influxdata/kapacitor/services/replay/service.go:1194 +0x124
github.com/influxdata/kapacitor/services/replay.(*Service).doReplayFromRecording(0xc4201f08f0, 0xc4207603c0, 0x24, 0xc420793ec0, 0xc42071ab10, 0x24, 0xc420788be0, 0x4a, 0x0, 0x11c, ...)
        /root/go/src/github.com/influxdata/kapacitor/services/replay/service.go:1121 +0x1ee
github.com/influxdata/kapacitor/services/replay.(*Service).handleCreateReplay.func1(0xc4201f08f0, 0xc420793a80, 0xc420793ec0, 0xc42071ab10, 0x24, 0xc420788be0, 0x4a, 0x0, 0x11c, 0xed13fe9c4, ...)
        /root/go/src/github.com/influxdata/kapacitor/services/replay/service.go:956 +0xbe
created by github.com/influxdata/kapacitor/services/replay.(*Service).handleCreateReplay
        /root/go/src/github.com/influxdata/kapacitor/services/replay/service.go:958 +0x528

[pshemk]

After commenting out '.post()' it works fine, so the issue is with '.post()'.

At a closer inspection I realised I have:

.post('http://nagios-api:8080/message')

with single quotes, when the docs here: https://docs.influxdata.com/kapacitor/v1.3/nodes/alert_node/ indicate that I should use double quotes, but with double quotes:

root@kapacitor:/tmp# kapacitor define load_alert -type stream -tick /var/lib/kapacitor/load-over1.tick -dbrp telegraf.autogen
invalid TICKscript: line 7 char 9: cannot assign *ast.ReferenceNode to type string, did you use double quotes instead of single quotes?

So I changed it to use 'endpoint()' and single quotes:

stream
     |from()
        .measurement('system')
        .where(lambda: "alerting_load" == 'over3')
     |alert()
       .crit(lambda: "load1" > 0.1)
       .post()
          .endpoint('http://nagios-api:8080/message')
       .log('/tmp/alerts.log')

but when I attempted to replay it - container died again in exactly the same way.

[MarcDenman]

I have just run into this as well, using the latest docker image I found that having .post() in my TICKscript caused the same error to occur.

I have ended up having to use the 1.2.1 container where the problem doesn't exist. I found the same issue with the 1.3 container.

[desa]

@pshemk @MarcDenman Just to clarify, the panic only occurs when using a replay, or does it also occur when using the task is running?

[MarcDenman]

@desa I only tried using a replay, for some reason I didn't even think about actually running the task and seeing if it happened "normally", sorry!

[desa]

@MarcDenman no problem, I'm sorry that it's panicking in the first place. It definitely shouldn't do that.

[pshemk]

root@kapacitor:/# curl -I http://localhost:9092/kapacitor/v1/ping
HTTP/1.1 204 No Content
Content-Type: application/json; charset=utf-8
Request-Id: db3769d7-aeb0-11e7-90f1-000000000000
X-Kapacitor-Version: 1.3.3
Date: Wed, 11 Oct 2017 18:20:27 GMT

[desa]

Awesome! Thanks. I've got a PR open with a fix. Just need to add some tests and then it should be good to go.