Blocks monitoring websocket disconnects when relay transaction processing takes too long

[nymkappa]

Hi everyone, I'm creating this discussion in relation to the following PR #61 by @canhlinh. Here is my understanding so far.

What is the issue?

When the relay spend to much time broadcasting transactions to the destination chain (_relay), the blocks monitoring websocket disconnects. This is maybe due to a buffer overflow in the websocket.

How did we come to this conclusion?

ICONDAO team provided this go test script.

In this script you can notice a <-time.After(time.Millisecond * 300) sleep between each received blocks.

When running the script as is, after a couple of minutes (~15 minutes in my own test) the websocket connection is terminated unexpectedly. By removing the sleep code, the script runs for hours without any interruptions. Which is why we think it's a buffer overflow issue.

What are the possible fixes?

• The issue was identified thanks to this pull request Keep websocket connection alive #61. While it seems that ICONDAO has now a stable connection with this code, it is not clear at all why this fixes the previously mentioned issue. Therefore we will most likely decline the PR unless a more detailed explanation of this fix comes up in this dicussions.
• Run monitoring and relaying in two different threads. According to this comment Keep websocket connection alive #61 (comment) the monitoring is blocked while the relay is sending transaction on the destination chain. We think that both things should not be blocking each others and should communicate via internal queues they can share.
• @jspark-icon mentioned about implementing network throttling on the BMC. @jspark-icon could you please provide some details about this idea?
• Automatic reconnection mecanism

Feel free to share your thoughts about this issue and propose some workarounds. If you believe our root cause analysis is invalid, please let us know about yours.

Thank you.

[nymkappa]

A possible explanation of this disconnection behavior is the following:

• When the BRM is relaying transactions through _relay function, the monitoring channel is blocked
• Because the monitoring channel is blocked, the websocket buffer on the server side is growing, until it is full.
• When the server side socket buffer is full, the server stop sending data and wait
• At this point there is no network activity on the connection anymore
• Since there is no network activity on the connection, the infrastructure may decide at some point to terminate the connection to free up resources

The PR mentioned earlier, would therefore actually keep the connection alive by reactivating it every 10 seconds. The problem is that it does not really solve the buffering problem from the server side. Having this keep alive system would add some unnecessary load on the infrastructure.

Idea to explore

We should limit the number of BTP messages currently processing between two chains at any point of time. So for example, if ICON <> PRA has limit of 10 BTP simultaneous messages between two specific BMC on the network, all extra attempt to send BTP messages should revert. Once one of the currently processing BTP messages has completed (both networks are in sync regarding the coin/token transfer status), a new BTP message request can be accepted.

This of course has to be discussed and fine tuned, and we hope you guys can share some feedback.

Thank you.

[canhlinh]

Hi @jorisvial Please let me know the code where the WebSocket is buffering?

From my view, At this line on the server-side. It points out that this code will block until the client reads a new message. There is no buffering here. Is it correct ?

[nymkappa]

The buffering mentioned happens at much lower level (OS API level or even network drivers level). I'm not familiar with TCP RFC so I can't tell where exactly this would happen, but it's not related to the go implementation.

[mksong76]

@canhlinh Besides the case, you may see connections lost for various reasons. When it's disconnected, it may continue to receive messages from the height of the next block of the last message. That's what we have designed to be used. So, you can easily recover the events stream without receiving old events.

With that reconnection mechanism, you don't need to add PING/PONG to keep it alive. And the reconnection mechanism is a mandatory feature for the recovery from network problems.