transport-manager: Too many file descriptors crash

[lexnv]

Issue

Litep2p crashes after 4/5 hours when tested with a high number of inbound / outbound connections (500 in Kusama).

2024-11-07 21:03:37.155 ERROR tokio-runtime-worker grandpa: GRANDPA voter error:
could not complete a round on disk:
Database error: IO error: While open a file for appending: ../dbs/2024-11-04_07-55-18-9e0cebad9f60d62c72c933340c8a3f9e3af63c76-litep2p/chains/ksmcc3/db/full/591987.log:

Too many open files   

Investigation

This may happen because both TCP and WebSocket transports are polling the listener socket which produces sockets (file descriptors) that await negotiation:

litep2p/src/transport/tcp/mod.rs

Lines 526 to 530 in af0535c

	impl Stream for TcpTransport {
	type Item = TransportEvent;
	fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
	if let Poll::Ready(event) = self.listener.poll_next_unpin(cx) {

The transport manager looks like its not consuming the events:

2024-11-07 20:13:39.554  INFO tokio-runtime-worker litep2p::websocket: pending_inbound_connections=0

2024-11-07 20:13:59.556  INFO tokio-runtime-worker litep2p::websocket: pending_inbound_connections=217

In the span of 20 seconds we received roughly 1653 pending inbound and handled 1436.

Possible Solutions

• Don't poll the socket listener (TCP and WebSocket) if we get above a number of pending_inbound_connections configurable by TcpConfig and WebSocketConfig
• Ensure TCP and WebSocket are robust wrt "too many open files" errors from the listener
• investigate further polling of futures (transports / transport-manager): it looks offhand like we were able to sustain this pace in the past

[lexnv]

The first graph highlights litep2p supporting many inbound connections when substrate is configured for 500/1k in peers.
The second graph shows a memory leak in our tracking of connections which is included in one of our latest releases.

This does not reproduce anymore. I suspect this was an artifact of having multiple polkadot workspaces opened with vscode and rustanalizer, while running 2 substrates nodes on the same machine.