Raphael/log errors

[raphaelgavache]

Problem

After a trace-agent crash /var/log/datadog/trace-agent.log did not contain the error.
In /var/log/datadog/supervisord.log you could find this error:

2017-05-22 20:15:01,832 INFO exited: trace-agent (exit status 2; not expected)
2017-05-22 20:15:02,482 INFO gave up: trace-agent entered FATAL state, too many start retries too quickly

And by running the trace agent manually you could see the goroutines spans.
More details on trello

Fix and test

The root of the problem is watchdog.Go() function : if you add a panic(...) in it https://github.com/DataDog/datadog-trace-agent/blob/master/watchdog/logonpanic.go#L49, you will reproduce the error.
Now if you go back to the old way (before factoring) of creating go routines

	go func() {
		defer watchdog.LogOnPanic()
 		server.Serve(stoppableListener)
	})
	}()

and add panics in it, the crash will properly log.

Did tests by running different versions of the trace-agent on one node of staging, adding panics and changing code.

Adding a panic at this exact line: https://github.com/DataDog/datadog-trace-agent/blob/master/model/statsraw.go#L211 reproduces the error explained in the trello card. Doing it with the reverts below will solve the issue (tested it and it does log)

[ufoot]

I'm very fine with reverting this watchdog.Go thingy, to be honest, I prefer the "manual" defer syntax, IMHO it's easier to understand what it does. Still, there's a conflict with master now (I'm the culprit, having changed some things in concentrator). Could you please resolve that conflict and re-submit the PR? This way I could really validate it. Thanks in advance.

[ufoot]

👍 thanks