reduce buffer size of "unused" side of socket

[gregcusack]

Follow Up to PR: #3929

Problem

We allocate large send and receive buffers for all sockets. However, not all sockets read and write equally from/to their buffers. In fact, 6 sockets are primarily Read and 2 are primarily Write. Meaning we are allocating memory that is never used.

Summary of Changes

1. TPU write side of sockets are only QUIC control traffic. Set buffer size to 4 MB
2. Broadcast and Repair read side of sockets are only QUIC control traffic. Set buffer size to $ MB

Services/Sockets
Read/Write

Gossip
RPC - TCP
Ip_echo - TCP
Tvu
tvu_quic
Repair
Repair_quic
Serve_repair
Serve_repair_quic
Ancestor_hashes_requests_quic
Ancestor_hashes_requests
Primarily Read

Tpu
Tpu_forwards
Tpu_vote
Tpu_quic
Tpu_forwards_quic
Tpu_vote_quic
Primarily Write

Retransmitter
Broadcast

Follow Up PR:

1. Make better sizing decisions for the sockets that don't need a full 128MB buffer

[steviez]

I know you haven't marked this PR for review yet, but seeing the repeated comments about the kernel doubling the buffer size, I'm wondering if we should make our function account for the doubling. Ie, a user passes X and we do X / 2 internally so that they actually get X

[gregcusack]

so i was thinking about that but i wanted to keep the same behavior as setsockopt syscall. thought it would be confusing to have a different behavior. then again, idk how many people are aware of what setsockopt is and how its implemented.

[steviez]

thought it would be confusing to have a different behavior

Yeah, that's a fair counter. Essentially, a question of should our API make things easier OR remain consistent with the underlying API.

then again, idk how many people are aware of what setsockopt is and how its implemented.

Ha yeah, this is the other angle of it. I had to crack man open, but looking at the notes:

$ man 7 socket | grep NOTES -A 2
NOTES
       Linux assumes that half of the send/receive buffer is used for internal kernel structures;
       thus the values in the corresponding /proc files are twice what can be observed on the wire.

Based on that, the extra half that Linux takes should not be considered available for user payloads.

All that being said, maybe it's better not do any division on our side and just pass the value through as you have it

[gregcusack]

oh i missed that. ok so if we set 4 MB, the kernel doubles to 8 MB, but only 4 MB are usable on the wire. So ya, let's not divide but let's set to 4 MB knowing kernel will increase it but only the original/desired 4 MB is usable