Pull Queries: include unique correlation id in processing logs

[big-andy-coates]

At the moment the pull query code builds a unique query id per request and embeds this in the result returned to the user.

We should extend this so that each request the rest server receives should generate a unique correlation id, which is logged at the start of the call. This correlation id should then be passed to ConfiguredStatement. For pull & transient queries, this id should be used in the query id, for anything persisted to the command topic this correlation id should be included.

This will allow people to use the logs to trace back when and who made the original request.

[AlanConfluent]

We should consider doing this using either a http://logback.qos.ch/manual/mdc.html, or just explicitly logging a generated id.

The only issue is that logging per pull query hurts performance, so we only do debug logging at the moment. This presumably would be a part of those logs.

[agavra]

That makes a lot of sense, it seems like log4j has a similar concept: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/MDC.html I hadn't heard of that before, but seems promising.

Given that the performance characteristics are questionable, I think we can probably give this a lower priority - but it should be on our radar to build out our support model. I'll leave the needs-triage for now, but I think p1 makes sense