nsqd: configurable queue scan worker pool

[andyxning]

This PR adds queue-scan-worker-pool-max command line argument.

In our case, the cluster is about 600+ channels and a defer channel with high qps, the phenomenon is that the defer channel message has a large e2e delay beyond the deferred time. The root reason is that for nsqd, there are at most 4 scan worker which is too small for our cluster.

After setting queue-scan-worker-pool-max to 64, the defer message delay is more accurate.

[andyxning]

/cc @jehiah @mreiferson @ploxiln

[ploxiln]

Good point.

I think that the description of the flag could be better, it is not really clear what it is for just from the description. But it's not really easy to come up with a concise description for this, it's sort of an implementation detail. I might say something like "max coroutines for managing in-flight and deferred queues".

By the way, because QueueScanSelectionCount is 20, at most 20 can do work at once, because queueScanLoop() sends 20 channels for work, then waits for them all to be done (then repeats). So 44 of your 64 queueScanWorkers are doing nothing. Maybe this should be adjustable as well? Maybe it should just automatically be set to worker-pool-max if worker-pool-max is greater?

I was thinking about basing these numbers on cpu cores available, which may be convenient for users that run nsqd on different sized servers so they don't have to configure it explicitly, e.g. a 2-core in development and a 32-core in production. But the user might not want nsqd to go crazy with workers just because it's in one of many containers on a large server. So just being configurable is a fine idea.

[andyxning]

I might say something like "max coroutines for managing in-flight and deferred queues".

Done

Maybe this should be adjustable as well? Maybe it should just automatically be set to worker-pool-max if worker-pool-max is greater?

But the user might not want nsqd to go crazy with workers just because it's in one of many containers on a large server. So just being configurable is a fine idea.

By allowing end users to customize seems more reasonable. What we should do is to make the description more readable and clear. :)

[ploxiln]

I re-worded the help description again, I hope that's OK.