Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MM-55415] Fix race condition on stop #124

Merged
merged 2 commits into from
Nov 17, 2023
Merged

[MM-55415] Fix race condition on stop #124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e12aacf
Fix race condition on stop
streamer45 Nov 17, 2023
876fb8e
Wait for signaling goroutines to exit before returning on close
streamer45 Nov 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion service/rtc/server_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -461,7 +461,7 @@ func TestCalls(t *testing.T) {
func TestTCPCandidates(t *testing.T) {
log, err := logger.New(logger.Config{
EnableConsole: true,
ConsoleLevel: "INFO",
ConsoleLevel: "DEBUG",
})
require.NoError(t, err)
defer func() {
Expand Down
1 change: 1 addition & 0 deletions service/rtc/session.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ type session struct {

closeCh chan struct{}
closeCb func() error
doneWg sync.WaitGroup

vadMonitor *vad.Monitor

Expand Down
25 changes: 23 additions & 2 deletions service/rtc/sfu.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,8 @@ func (s *Server) InitSession(cfg SessionConfig, closeCb func() error) error {
us.initBWEstimator(<-bwEstimatorCh)

peerConn.OnICECandidate(func(candidate *webrtc.ICECandidate) {
us.mut.RLock()
defer us.mut.RUnlock()
if candidate == nil {
return
}
Expand All @@ -252,6 +254,14 @@ func (s *Server) InitSession(cfg SessionConfig, closeCb func() error) error {
s.log.Error("failed to create ICE message", mlog.Err(err), mlog.String("sessionID", cfg.SessionID))
return
}

select {
case <-us.closeCh:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we're just checking if the session has been closed before writing to its receiveCh? (just restating for my own understanding)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but we do it under lock (us.mut), that's the key essentially. So we are guaranteeing that either closeCh has been closed already when we are inside this block and we exit early otherwise s.receiveCh must be still open and we proceed. Unfortunately I couldn't merge the two selects because the order of selection is not deterministic so we could still trigger a race.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it's random on purpose. Gotcha thanks!

s.log.Debug("closeCh closed during ICE gathering", mlog.Any("sessionCfg", us.cfg))
return
default:
}

select {
case s.receiveCh <- msg:
default:
Expand Down Expand Up @@ -550,7 +560,9 @@ func (s *Server) InitSession(cfg SessionConfig, closeCb func() error) error {
}
})

us.doneWg.Add(1)
go func() {
defer us.doneWg.Done()
select {
case offer, ok := <-us.sdpOfferInCh:
if !ok {
Expand All @@ -573,7 +585,11 @@ func (s *Server) InitSession(cfg SessionConfig, closeCb func() error) error {
return
}

go us.handleICE(s.metrics)
us.doneWg.Add(1)
go func() {
defer us.doneWg.Done()
us.handleICE(s.metrics)
}()

s.handleTracks(call, us)
}()
Expand Down Expand Up @@ -629,8 +645,13 @@ func (s *Server) CloseSession(sessionID string) error {
}
call.mut.Unlock()

us.rtcConn.Close()
us.mut.Lock()
close(us.closeCh)
us.mut.Unlock()
us.rtcConn.Close()

// Wait for the signaling goroutines to be done.
us.doneWg.Wait()

if us.closeCb != nil {
return us.closeCb()
Expand Down