quic: make the reuseport feature optional

[gfanton]

Hi from @berty,
We recently discovered that if we enable the reuseport feature on the libp2p-tcp-transport while using a cellular connection, it could lead to network errors with some mobile operators. This is the case with the French operator Free Mobile which is massively used in France and we suspect that the problem occurs with several other operators around the world.

It seems that the QUIC transport has exactly the same issue but does not have an option to disable the reuseport. At the moment, this prevents us from using the QUIC transport on cellular connection even though it is particularly suitable for mobile use.

It would be nice to make the feature optional like in the libp2p-tcp-transport. I checked if it was possible to simply add an option to be able to toggle it off, but the feature seems too tightly bonded to the rest of the code.

[marten-seemann]

That would imply that they've built special logic to identify QUIC connections. QUIC connections rotate their Connection IDs on a regular basis to make exactly that more difficult.
What's the error you're seeing? Do you have any qlog traces?

[gfanton]

The main reason it took us a while to figure out the cause is that no explicit error is raised since the error actually occurs on the operator's side. The only thing we can notice as a user of their network is that a libp2p node with go-libp2p-tcp-transport and reuseport feature enabled, once it has opened several connections, will starts hanging on endlessly without being able to read on their connections.

The same behavior seems to occurs with go-libp2p-quic-transport, so I don't think it's a QUIC related problem, but rather something on the operator side: their network seems unable for some reason to communicate with different peers using the same port.

To demonstrate this, I wrote a simple golang script that starts several hosts on a remote server and a single host on the local machine. The latter then starts pinging each remote host in a loop and if a ping takes more than 10 seconds to complete, the connection is considered dead. The test succeeds if after a given period (e.g. 1 minute) all connections are still active.

When the test is run on a Free Mobile cellular connection using tcp-transport with reuseport enabled or quic-transport, all connections will eventually enter a dead state. And the more connections that use the same port, the faster they die. However, the test always succeeds in all other cases:

• On a Free Mobile cellular connection using a tcp-transport with reuseport disabled.
• On a Free Mobile cellular connection using a tcp-transport with reuseport enabled or a quic-transport BUT using a VPN, so only one connection from the point of view of the Free Mobile network.
• On any other connection tested so far (cellular or not) using quic-transport or tcp-transport whether the reuseport option is enabled or not.

You can find the script here: https://github.com/gfanton/libp2p-reuseport-test
I wrote another golang script that only use the TCP standard library (without any libp2p import) which produces exactly the same behavior as described above: https://gist.github.com/gfanton/83ea66cb1dfaa7c5a6c623c288d52fa4

I tried to find relevant information in the qlog traces using QLOGDIR without success since it will only hang on read. If you need to do further tests, we can provide you with an ssh endpoint on a machine connected to the internet via a Free Mobile cellular connection.

[marten-seemann]

Interesting. And an impressive amount of research to locate the problem. Have you ever tried getting in touch with Free Mobile? This very much sounds like a problem on their side, and they might be able to fix it for good.

Regarding disabling reuseport in this repo, I think that shouldn't be too hard. It would probably (haven't tried it yet myself though) be enough to just use quic.Dial here: https://github.com/libp2p/go-libp2p-quic-transport/blob/36ab344ec0a4b2cb17b2a0bbd7904b357aaa0bc6/transport.go#L91-L97. Contributions welcome!