Flask SocketIO memory leak and WebSocket Breaks frequently (During high frequency of data emission)

[VipulRJ]

Hello Miguel,
Thanks for continued support. But this time, i again had observations of memory leak. Please refer videos for better clarity.

I have below Flask SocketIO-2.0 setup.
1.Flask-SocketIO-2.0.tar.gz
2.python-socketio-1.0.tar.gz
3.python-engineio-0.8.7.tar.gz

And running yours sample example of Flask-socketIO-2.0
Flask-SocketIO-master.zip
To run sample of Flask-SocketIO-2.0, please install this - gevent-socketio-0.3.6.tar.gz

I had below observations using Google Chrome. I have uploaded the video for your reference.
Step-1: At Initial, (Refer video ‘01 InitWebsocket.mp4’ at url https://www.dropbox.com/l/s/UM2SlTS2Ep7V6cvtLK3udu

1. UI html page is able to initiate web-socket connection with server.
2. Web-Socket connection is having status code: 101 with Time status as Pending. Also, data is visible in Frame section of browser’s debug window
3. Memory consumption started with 0.6% of system memory
4. Date size is 21 Kbyte
5. Data emit rate is time.sleep(0.05). Data-emit rate is kept high just to reproduce issue fast enough rather waiting for issue to reproduce post couple of days.
6. At initial - No memory leak observed

Step-2: After few minutes i.e. Approx. 3 minutes, (Refer video ‘02 WebSocketMemoryLeak.mp4’ at url https://www.dropbox.com/l/s/MQRACKo8qdv8elUPjNckQo

1. Data display on browser is slowed down. It can be visible in Frame section of browsers debug window
2. Gradually, memory consumption starts increasing and reaches to approx. 2.2%, followed by web-socket connection breaks/completes at 8.3 min.
3. On web-socket connection failure, Client fall backs to polling mechanism. Some Assertion errors noticed on terminal. Client re-attempts for web-socket connection. On web-socket re-connection, some messages as 2probe, 3probe, 5 appear. This messages may be internal to web-socket library.
   Memory consumption is still 2.2%
4. One thing noticed that “while web-socket connection broke, some data packets were missed” to be display on screen and even do not appeared in polling response to UI
5. From now onwards, whenever web-socket connection breaks, Opcode -1 displays in Frame section. ‘Client disconnection’ message appears on terminal. Memory consumption increases by approx. 1%. Web-Socket connection persists only either for seconds or for <2min.
6. Point. 5 reproduces continuously. Memory consumption increases by approx. 1%.

Step-3: Actually, in step-2, web-socket connection was able to establish. But now in step-3, I would make changes in browser settings to block web-socket connection permanently and instead let polling mechanism
only occur frequently. This change is made to notice that Memory Leak also happens heavily during polling and while closing the browser and re-opening the browser page.
Refer video 03 PollingMemLeak.mp4 at url https://www.dropbox.com/l/s/urnBdMkJqjt0aEcUHJCahr

1. Memory consumption is already at 21%
2. While polling was in progress, memory leak increased from 21% to 22%. Later, I refreshed the screen, memory increased from 22% to 23.6%. Suddenly, 400 BAD Request and 500 Internal Server error appeared on browser’s debug window, some key error appeared on terminal and memory reduced from 23.6% to 23%.
3. Again memory consumption started increasing from 23% to 24.2%, Before, internal server error appears and drops to 23%, I quickly again refreshed the screen, memory consumption started increasing from 24.2% to 25.2%. On again refresh, memory from 25.2% to 25.6%.

Observations/Summary for Step2 and Step3:
Step-2:

1. Whenever approx. 7000 to 8000 data packets pushed to browser, later data appearing on browser’s frame window get slower followed by memory leak and ending up with Web-Socket connection failure. This caused to re-attempt web-socket connection.
2. Frequent web-socket connection breakages occur.

Step-3

1. In polling mechanism, memory consumption keeps increasing by 1%. The moment 404 and 500 error appears, memory drops by 1%.
2. Before 404 and 500 error appears, if user refreshes screen, then memory never drops and instead increases by another 1%

I am looking for below options

1. Since polling mechanism is one of the major factor causing memory leak, Is there a way to disable the Polling mechanism? I am fine if data does not emit to browser than to have memory leak.
2. If you know any other proven WebSocket library then let do us know. I am ready to procure the proven one.

I have posted this issue on your blog - discussion thread #326, #327

Thanks & Regards,
Jain

[miguelgrinberg]

You don't seem to be using the original example, since in your videos the events go at a much higher rate. Can you share your version of the code?

[VipulRJ]

Hello,
I am using the original sample source ( https://github.com/miguelgrinberg/Flask-SocketIO/tree/master/example).

As said earlier in Step.1 point.4 & 5, I have only changed the data content and its size is changed to 21KB. Most importantly, I changed Data emit rate to time.sleep(0.05). Here, Data-emit rate is kept high just to reproduce issue fast enough rather waiting for issue to reproduce post couple of days.

Where code version of sample example be visible? I have shared the version details of packages.

Regards,
Vipul Jain

[VipulRJ]

Hello Miguel,
I have below packages.
1.Flask-SocketIO-2.0.tar.gz
2.python-socketio-1.0.tar.gz
3.python-engineio-0.8.7.tar.gz

And running yours sample example of Flask-socketIO-2.0 "Flask-SocketIO-master.zip"
Of this attachment, i am using only file app.py and index.html of directory 'example'.

I just did a diff between my example copy and latest example copy available is changes in directory 'flask_socketio', test_socketio.py and setup.py. This changes were done a day ago i suppose. Anyway, i am not using these files.

Let me know if i need to provide additional info.

Regards,
Vipul Jain

[miguelgrinberg]

You described the changes you made to the script, but you haven't provided them. It would be best if you give me the exact script you are using, instead of me trying to figure out the changes on my own. Just commit them to your fork of the project and I'll get the files from there. Thanks.

[VipulRJ]

As request, please find attached example.zip along with below changes. Changes are in app.py, and index.html. New additions are static directory and there-in js files. Changes are tagged with 'vipuljain'
example
example.zip

-> app.py 

templates
-> index.html
static
-> jquery-1.4.2.min.js
-> socket.io.min.js

Let me know if you are able to download the attachment and run the sample test.

[VipulRJ]

I suppose the directory listing was disturbed, updated again as below.
example
-> app.py
templates
-> index.html
static
-> jquery-1.4.2.min.js
-> socket.io.min.js

[miguelgrinberg]

Well, using your application I don't really see continuously growing use of memory. On my system memory kept growing for about 4 minutes, then stabilized and stayed the same for a long time. I think what you are seeing is the Python's not terribly effective memory management of those large strings you are creating. Note that I was able to make the process consume less memory by storing the result of json.dumps() in a variable instead of regenerating the json string every time.

[miguelgrinberg]

Update: I left the script running for over an hour and I don't see any change in memory usage.

[VipulRJ]

Hello Miguel,
I accept your suggestion regarding storing the result of json.dumps() in a variable instead of regenerating the json string every time.
Post incorporating above suggested changes, still memory leak continues. Attached images for quick reference.

I had captured observations using Google chrome Version 48.0.2564.109m with app.py running on python2.7 base platform on Debian linux.
Same i tried with browser Mozilla v37.0.2, memory did not grow-up but Mozilla crashed. I am not getting the root cause of memory leak. There may be multiple sources contributing to memory leaks in server side.

I would like to replicate your setup details at my end if you can share your setup details where you ran app.py application.
Setup details like python ver, browser type and ver, OS-type, etc.

One thing i would like to know - were you able to see JSON data on browser continuously even after an hour. Because, by that time the browser memory may have crossed 1GB i assume.
Were websocket connection never broke? Were single web-socket connection (status code-101) continued for an hour?

I would request you to see attached videos once again and try to observe detailed step-2 and step-3 (as mentioned in earlier post).

1. Memory increase is one issue
2. Memory fall is different behavior but memory usage falls because of 400 BAD Request and 500 Internal Server error, followed by reattempt for connection

In addition, you also try - do not let web-socket connection establish and instead transport polling will take care. Observe memory leaks alongwith data to be observed on browser.

Regards,
Vipul Jain
Images.zip

[miguelgrinberg]

If I'm not mistaken, the entire period covered in your last screenshots is about two minutes. As I said above, in my testing it took about 4-5 minutes for memory to reach a balance.

Also, to avoid Chrome using too much memory you can comment out the code that adds those payloads to the DOM. Then Chrome will not use much memory and can stay up for a long period of time.

[miguelgrinberg]

I gave it another run today, and I still don't see any evidence of memory leaks.

• I'm using only polling, as you suggested
• I'm not getting any 400/500 errors, all requests are suceeding (Chrome's network tab shows all requests return 200)
• I'm not dumping that huge payload to the page, to avoid Chrome spending lots of memory. This allows the test to run for longer.
• The server caches the output of json.dumps, that helps reduce garbage collection effort by the interpreter.
• I'm running the test on Mac OS X.
• The server process uses about 17MB of RAM. This current test has been running for about 12 minutes now, it's been at 17MB for most of the time. The total system memory stats also remain more or less constant.

[miguelgrinberg]

This issue felt through the cracks. A while ago I have addressed a newly discovered socket leak that is probably what is reported here. Please upgrade all dependencies to the latest versions and retest. Reopen if you still have leaks.