transport: write buffer size should equal the size specified by the user

[easwars]

This issues addresses bullet point 1 from #5751.

Write buffer sizes can be configured on the client and the server. The docstring on these two APIs clearly states:

The corresponding memory allocation for this buffer will be twice the size to keep syscalls low.

Looking at the implementation though, it is not obvious what benefit we get out of configuring a buffer which is twice the size.

Once we have addressed #5757, we can see if there is any change in benchmark results as a result of using a buffer which is the same size as configured by the user.

Another possibly important data point is that the buffer is flushed not only when we have buffered more than the user configured size, but also when the loopy writer decides that it should be flushed. It is important to understand why this is being done. See:

grpc-go/internal/transport/controlbuf.go

Line 575 in 0d6481f

l.framer.writer.Flush()

[atollena]

Another possibly important data point is that the buffer is flushed not only when we have buffered more than the user configured size, but also when the loopy writer decides that it should be flushed. It is important to understand why this is being done.

This call to Flush() is necessary to make sure that we eventually write data on the transport when the buffer is not filled entirely. So each time there is data available and loopy runs, it gets a chance to flush the buffer.

[atollena]

I ran some benchmark and put the results alongside a proposed change in this pull request: #5807.

[atollena]

This was fixed in #6983.