Race Condition in Network Package Causes Gossamer to Crash

[kishansagathiya]

Describe the bug

• While running a devnet, one or more nodes shut down because of runtime: unknown pc error.

Expected Behavior

Current Behavior

Possible Solution

To Reproduce

Steps to reproduce the behaviour:

• I tried this with a 4 node devnet, with all of them as authorities and alice being babe lead.
• Wait for say five minutes, some of those node would shut down
  ( Look at https://hackmd.io/@nZ-twauPRISEa6G9zg3XRw/B1RFO3uDt for devnet setup)

Log output

• Attaching logs for all 4 nodes here. In this particular case, alice, bob and charlie shuts down on their own while, I shut down dave.
  https://gist.github.com/kishansagathiya/395cb914ce338e6081a73d58c51527f6

Specification

• go version:
• gossamer version:
• gossamer commit tag:
• gossamer commit hash:
• operating system:
• additional links:

[qdm12]

The problem seems to come from 5843324 as far as a few minutes running Gossamer was done for each commit from 86624cf down to a few commits. I'm still testing to verify this is right, and will make a fix PR for that commit.

[kishansagathiya]

I had a feeling that it could be because of a race condition is get functions in dot/network/peersdata.go.

For example,

gossamer/dot/network/peersdata.go

Lines 35 to 39 in 5843324

	func (p *peersData) getMutex(peerID peer.ID) *sync.Mutex {
	p.mutexesMu.RLock()
	defer p.mutexesMu.RUnlock()
	return p.mutexes[peerID]
	}

I think differs runs before return statements. (https://www.digitalocean.com/community/tutorials/understanding-defer-in-go, https://medium.com/a-journey-with-go/go-how-does-defer-statement-work-1a9492689b6e)

So, we are locking and unlocking before the task interesting to us. We should do something like this

 func (p *peersData) getMutex(peerID peer.ID) *sync.Mutex {
        p.mutexesMu.RLock()
        defer p.mutexesMu.RUnlock()
-       return p.mutexes[peerID]
+
+       mtx := p.mutexes[peerID]
+       return mtx
 }

Same goes for getInboundHandshakeData, getOutboundHandshakeData.

I ran the network after fixing this, but I still saw the problem. But this does seem inappropriate and we should fix this.

[kishansagathiya]

closed in #2393

[qdm12]

Fixed by #2408