Gracefully handle panics in all request handlers

[dnephin]

http.Server handles panics in request handler goroutines using recover, but other servers used by Consul do not. By handling panics in request handlers we will prevent any panics in request handlers from shutting down the entire process.

• HTTP API should already work correctly because http.Server handles panics
• grpc, xds: recovery middleware to return and log error in case of panic #10895 XDS grpc.Server can use grpc-middleware/recover (or an interceptor with a similar implementation)
• grpc, xds: recovery middleware to return and log error in case of panic #10895 Server RPC grpc.Server can use the same
• the Server port has many services behind it. Server.handleConsulConn can recover from most, but we should also look at Server.handleSnapshotConn and Server.handleInsecureConn and probably do the same there.
• DNS interface - unknown. needs investigation

[gmichelo]

Concerning grpc.Server, what do we want to do with the panic? Just turn it into a codes.Internal error? http.Server also prints the stack trace to stderr. If stack trace is not needed, something like the following should suffice:

gmichelo@2b14a9b

This is a prototype, if this approach is confirmed, I plan to add comments, tests and perhaps try to find a way to make the middleware registration common in a shared function.

[dnephin]

Thank you for working on this!

Returning codes.Internal and printing the stack trace using logger.Warn (or logger.Error) sounds like a good way of handling a panic. That diff also looks like a good start!