fix(server): Adjust batching behavior to reduce network latency on MULTI blocks

[royjacobson]

1. Add a Yield() call before executing the last command in the async queue when needed.
2. Allow the receive buffer to grow when needed.
3. Improve debugging logs for batching behavior.

The whole 'triggering the bug' process looks like this:

1. We process a larger packet with multiple short commands inside a MULTI segment.
2. We read the commands into a relatively small buffer (size 256)
3. The ParseRedis loop reads commands from the first part of the packet and pushes them into the dispatch queue.
4. The dispatch fiber processes all the commands in the dispatch queue quickly and without preemption because it's inside a MULTI block and all commands return immediately. After emptying the queue it calls SetBatchMode(false) which causes a flush of the response buffer after the next command is executed.
5. The rest of the commands are parsed and executed, but the second packet is not sent until an ACK is received from the client because of Nagle's algorithm.

Close #1285

[romange]

a small correction: SetBatchMode is not flushing anything but the subsequent reply will indeed flush.

Can you please add here how you reproduced the scenario with increased latency?

[royjacobson]

I reproduced it by running the provided benchmark from the issue (addr=redis://10.0.101.178:6378/7 go test -count=1 -v ./pkg/meta/... -run=TestDgfAndRedis).
The latency bug manifests after running it twice. (Probably due to some offset randomness or something?)

[dranikpg]

With memtier_benchmark --pipeline=2 you'll yield a lot 🤔 But its still more effective, right?

[dranikpg]

It would also be more efficient to check yield only if we didn't yield on the previous iteration (i.e. the transaction was run with inline scheduling)

[romange]

yes, we need helio machinery for this. Moreover, iouring api allows learning if there is more data pending in the network buffer pending after the last Recv call (IORING_CQE_F_SOCK_NONEMPTY from 5.19). I do not think I implemented support for this.

[royjacobson]

It does trigger quite a few more Yields, I'm not sure what's the performance impact

[royjacobson]

Updated to use the new helio epoch interface. Looked at the logs with memtier_benchmark and the JuiceFS benchmark and it looks quite good, probably 1 yield every 10 commands or something.

[romange]

can it be that we wake up on non-empty queue and it does not have size > 1 ?
I thought we only push to dispatch_q if we have more than 1 request.

[dranikpg]

At least all pubsub/monitor messages go over the dispatch queue and there we can wake for a single element