Possible crash when a client disconnects

[coolacid]

We've been using the websocket as a remote tool to change scenes and get information on current status. This is installed using the zip file.

We coded a custom web application to access the websocket, following the API documentations using the node API library.

The remote control web-app stays connected for hours at a time, and we've had it crash [consistently] after refreshing.

Crash dump:

Unhandled exception: c0000005
Date/Time: 2016-12-12, 00:04:11
Fault address: 545001F3 (c:\program files (x86)\obs-studio\bin\64bit\qt5core.dll)
libobs version: 0.16.6
Windows version: 6.3 build 9600 (revision: 17415; 64-bit)
CPU: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz


Thread 31E4 (Crashed)
Stack            EIP              Arg0             Arg1             Arg2             Arg3             Address
000000373FDCB810 00000000545001F3 00000037528F0870 000000000000000C 000000000000000C 0000000000000000 qt5core.dll!0x545001f3
000000373FDCB900 00007FFF72DC1274 00000037528F0870 000000373FE9EE40 0000000000000000 000000375277A490 qt5websockets.dll!0x7fff72dc1274
000000373FDCB940 0000000054505907 000000373FEB52E0 000000373FEBAD40 00000037528F0870 0000003752895130 qt5core.dll!0x54505907
000000373FDCBAF0 00000000548A7CA0 000000373FE9EE10 000000373FDCBC20 0000003752895130 00000000542A2C6E qt5widgets.dll!0x548a7ca0
000000373FDCBB20 00000000548A6BFA 000000373FDCFB40 000000373FE9EE10 0000003752895130 00000037528F0800 qt5widgets.dll!0x548a6bfa
000000373FDCC260 00000000544E23C9 00007FFF935545FA 0000000000000000 0000003752895130 000000373FDCC368 qt5core.dll!0x544e23c9
000000373FDCC2E0 00000000544E4161 000000373FE9EE40 0000000000000000 000000373FEBF3D0 000000373FDCC390 qt5core.dll!0x544e4161
000000373FDCC390 00007FFF78916F7F 0000003700000024 0000000000000000 0000000000000401 000000373FDCC598 qwindows.dll!0x7fff78916f7f
000000373FDCC3C0 0000000054527B25 0000000000000401 000000373FDCC598 000000373FDCC5C8 00000000000306C4 qt5core.dll!0x54527b25
000000373FDCC4D0 00007FFF935524FD 000000374044B690 000000373FDCC660 00000000000306C4 000002400350CFE6 user32.dll!0x7fff935524fd
000000373FDCC5A0 00007FFF93552357 00000000060003FF 0000000006000301 0000000000000000 0000000000000000 user32.dll!0x7fff93552357
000000373FDCC620 0000000054527299 0000000000000000 0000003700000024 0000000000000000 000000373FDCFB40 qt5core.dll!0x54527299
000000373FDCF9B0 00007FFF78916F59 000000373FEB52E0 0000003700000014 000000373FDCFA88 000000373FE9EE10 qwindows.dll!0x7fff78916f59
000000373FDCF9E0 00000000544DEDA1 000000373FEBB960 0000003700000000 000000373FE9EE10 00007FF7C1C043F0 qt5core.dll!0x544deda1
000000373FDCFA60 00000000544E1237 000000373FEAAE40 00007FF7C1C043F0 000000373FDCFD30 000000373FEA2F20 qt5core.dll!0x544e1237
000000373FDCFAD0 00007FF7C1AF6833 0000000000000000 0000000000000001 0000003700000000 0000000000000000 obs64.exe!run_program+0x1a3
000000373FDCFC30 00007FF7C1AF862E 0000000000000001 0000000000000000 000000373FDCFC68 000000373FEA9DE0 obs64.exe!main+0x35e
000000373FDCFDB0 00007FF7C1BE2195 000000373FE92938 0000000000000000 0000000000000001 0000000000000000 obs64.exe!WinMain+0x155
000000373FDCFE40 00007FF7C1BE1BA5 00007FF7C1BE1A48 00007FF7C13CF000 0000000000000000 0000000000000000 obs64.exe!__tmainCRTStartup+0x149
000000373FDCFE80 00007FFF913D13D2 00007FFF913D13B0 0000000000000000 0000000000000000 0000000000000000 kernel32.dll!0x7fff913d13d2
000000373FDCFEB0 00007FFF938A54E4 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ntdll.dll!0x7fff938a54e4

[coolacid]

I was able to reproduce it by refreshing a page rapidly.

Multiple refreshes slowly works fine, however, refreshing quickly enough causes OBS to crash.

[coolacid]

21:38:02.451: [obs-websockets] new client connected from �:40388
21:38:02.452: [obs-websockets] new client connected from ÿÿÿÿ:0

[Palakis]

@coolacid What you're telling is that several successive connections and disconnections to obs-websocket make OBS crash ?

[Palakis]

@coolacid Regarding the log excerpt in your last comment, this is a problem I already noticed. Can't tell right now if it's linked with this issue.

[coolacid]

Hey @Palakis I'm not sure if it's related either. Yes. Successive connections/disconnects will crash OBS.
At first we had multiple people connecting to our OBS, in order to narrow down the cause, I installed on a local laptop and ran the app locally.

At first I could only trigger it by refreshing multiple screens, I was finally able to consistently cause the issue only while refreshing semi-slowly.

Judging from the new client issue (and not knowing the underlying libs) is it possible a race condition exists between when the client connects, disconnects and the log line is logged?

Ie:

• Client connects, and quickly disconnects before hitting https://github.com/Palakis/obs-websocket/blob/master/WSRequestHandler.cpp#L57, The pointer could be nulled out at this point.

Note, I created a simple python app to connect to the OBS server and could not get a crash to happen. No matter how often I ran the script. This lead credence to the above since the python libraries ensure the connection fully happens before moving on. I'm trying to create something to consistently trigger the issue.

[coolacid]

At this point, I can confirm that the ÿÿÿÿ:0 appears to be related. It's also related to the browser closing the connection after sending the headers. While I can't get a script to trigger it, I tested again in the browser inspecting the WS connections. I can see the headers from the browser, however I triggered a refresh before the browser got the response, yielding the incorrect IP as above, and the crash on OBS.

[Palakis]

I've had this string pointer issue since the beginning of the project. I also had it in the Websocket message handler code but managed to solve it.
I've never looked back to it, but today I took a deeper look at the Qt docs and maybe a call to *QString::constData() is what I need to solve this.

[coolacid]

I just got two back to back crashes with the following trace:

Thread 464 (Crashed)
Stack EIP Arg0 Arg1 Arg2 Arg3 Address
00CFD70C 6B2EC9B2 00001178 09D1C540 09C6AB40 09C6AB40 obs.dll!obs_data_destroy+0x12
00CFD71C 63D13A8B 09C6AB40 00D0C738 6B4D27B6 00000001 obs-websocket.dll!0x63d13a8b
00CFD734 63D13C4B 00000001 01C4BE51 09C6AB40 00D31378 obs-websocket.dll!0x63d13c4b
00CFD740 6B4D27B6 00000000 00D30A58 00650030 00000000 qt5core.dll!0x6b4d27b6
00D0C740 00CFFBEC 00D30A58 00650030 00000000 00000000 !0xcffbec

Is it possible I'm killing the connection just in time where the plugin is still building the server connection side, but hasn't ran the kill connection code?

Maybe checking the socket is valid before sending data? Sorry, doing the best I can since I'm not a strong C coder ;)

[Palakis]

I can't reproduce the issue (tried with 64bit OBS Studio), but I anyway fixed the IP address issue in the log messages (27ec094).
Can you please test if crashes still happen with the latest codebase ? Here's a Windows binary build of it : https://transfer.sh/vfGCA/obs-websocket-27ec094-windows.zip

[coolacid]

Confirmed the logging issue is gone, however I can still cause OBS to crash. :(

JS code to crash

var ws = new OBSWebSocket();
ws.connect('10.0.4.153');

Refreshing at fairly fast random refreshes causes the crash.

[Palakis]

Where does the JS module you're using comes from ?

[coolacid]

https://github.com/haganbmj/obs-websocket-js

[Palakis]

Ha ! Now I can reproduce it !

[coolacid]

I don't think it's reconnections. I can refresh slowly and never hit the crash. However, if I catch the refresh just right the websocket never fully connects ('finished' status instead of 'switching protocols') which causes the crash.

[Palakis]

Agree, I can see this with the Visual Studio debugger attached. The pointer exception happens in a function call used by a function named handshakeReceived. Gonna investigate this issue further to see if it's a bug with Qt's WebSocket module and if I can do something about it in my code.

[haganbmj]

Not sure if it cares about whether its the same client reconnecting or multiple clients connecting in rapid succession. I can set up some tests this evening too.

[haganbmj]

Made a dozen connections within a second. No crash.
Tried to close all of those connections at the same time and OBS crashed.

Crash log.
http://pastebin.com/raw/27NRZe9k

Here's a simple test for this though. Download the page and run locally if you're on Firefox, otherwise I think Chrome and IE might allow insecure socket connections.
https://haganbmj.github.io/

EDIT: 25 Connections, 500ms between each, disconnect 250ms after connect.
Works for the first 12 connections, then crashes obs.

[Hardy535]

I also use obs-websocket-js and I get crashes more or less often.
It is everytime the same: Unhandled exception: c0000005
But I can't tell what I do to get it crashed.
//EDIT: Reconnecting over the web API is the most frequent reason.

Here is an example crash-log
I got enough of these (over 10) so if there is the need for I'll post them...

[Palakis]

Qt 5.8 will be available soon. Will test obs-websocket with it as soon as it's out.

[Palakis]

I notice that on the first ever connection to the Websocket server after launch, the framerate drops significantly.

[Theoretical]

https://github.com/Palakis/obs-websocket/blob/master/WSServer.cpp#L85-L86

The issue could simply be you're using a QList without using a QMutex. You're reading/writing from separate threads. Locking/unlocking on adding and removing a client is a must.

[haganbmj]

I'm gonna try messing with that later. Makes a ton of sense now that you say it, Theoretical.

[Palakis]

I found two different crash causes :

• Sometimes when deleting an instance of WSRequestHandler (commenting this line make most if not all crashes disappear)
• Sometimes when the destructor of WSRequestHandler is called and does something with the messageMap variable. (crash log)

Plus the concurrent access issue mentioned by @Theoretical (fixed in 748b6f6). I can't trigger a crash caused by this issue, even though theory says accesses to the clients list has to be protected with a mutex.

When testing, I use haganbmj's tool with the following settings :

• Number of connections : 1000
• Delay between each connection : 100 ms
• Disconnect : Yes
• Delay before each disconnect : 2000 ms

One can reproduce the first issue mentioned above with these settings.

[Theoretical]

https://github.com/Palakis/obs-websocket/blob/master/WSRequestHandler.cpp#L109

You're also deleting the object twice. Should be just lowered to once for that.

[Palakis]

@Theoretical "You're [...] deleting the object twice"
The QWebSocket _client object has no parent defined (it's default behavior), and deleteLater for this object is only called in the instructions block you mentioned. Where/when do you think _client is also deleted ?

[Theoretical]

https://github.com/Palakis/obs-websocket/blob/master/WSRequestHandler.cpp#L109
https://github.com/Palakis/obs-websocket/blob/master/WSServer.cpp#L99

Those are the same client objects being deleted at the same time, right?

[Palakis]

@Theoretical nope. Those are two different objects. _client is a QWebSocket attribute of WSRequestHandler and pClient is a WSRequestHandler object (its name is misleading and has been corrected to pHandler in my local copy).

[Palakis]

I finally managed to fix this issue by rewriting the way the plugin handles session-specific variables (e.g. : client authenticated or not) because I think I was misusing Qt's object model and event loop, which was the cause of several crash types I encountered while debugging. The messageMap mentioned above is also fixed.

Thanks to all the people who participated in this thread and in the debugging process. I spent a lot of time working on this issue and I'm glad it's fixed now.

[Palakis]

I identified an issue with authentication caused by the latest changes. Working on it right now.
Update: nevermind, false alert.