Socket.io memory doesn't drop of after clients disconnect

[94cstyles]

I use the most simple example

var app = require('http').createServer(handler)
var io = require('socket.io')(app);
var fs = require('fs');

app.listen(1994);

function handler(req, res) {
    res.end('socket.io start')
}

io.on('connection', function(socket) {});

I use socket.io-client, create a 4k connections
Then I closed the client, find memory doesn't drop.

socket.on('disconnect', funtion() {
    delete io.sockets[socket.id];
    delete io.sockets.sockets[socket.id];
});

heapUsed decreased slightly
rss and heapTotal without any change

SOS

The memory is cleaned up by the garbage collectors periodically.
But I waited for half an hour, still no change.
Can I only use global.gc() to forcibly reclaim memory it?

@nuclearace

[94cstyles]

It seems that once a client connects, it never gets deleted from socket.io , or socket.io itself a memory leak problem .

[tsuz]

@TOP-Chao I was stress testing socket.io and realized that issue as well.

After investigating, it seems that when global.gc() is called, it reduces the memory.
Although, it seems the memory seems to get reduced slowly.

The first image is not using global.gc()

The second image is using global.gc() every minute

The code:

server.js

var http = require('http');
var io = require('socket.io');
const PORT = 3000;
var process = require('process');
var server = http.createServer(function(req, res){
    res.writeHead(404, {'Content-Type': 'text/html'});
    res.end('<h1>Aw, snap! 404</h1>');
});

server.listen(PORT);
io = io.listen(server);

var connectedCount = 0;
console.log('start');

logMemory();

// Add a connect listener
io.on('connection', function(socket) {

    //console.log('Client connected.', connectedCount);
    connectedCount++;

    // Disconnect listener
    socket.on('disconnect', function() {
        connectedCount--;
//        console.log('Client disconnected. Total Connections: ', connectedCount);
        if(!connectedCount) {
            logMemory();
        }
    });

});

setInterval(function(){
    logMemory();
}, 60000);

function logMemory() {
    var memory = process.memoryUsage();
    console.log((new Date()), 'RSS ',memory.rss, ', Heap Used ', memory.heapUsed, ' Heap Total ', memory.heapTotal);
}

client.js

var io = require('socket.io-client');
const CLIENT_COUNT = 100;
const PORT = 3000;

for(var i = 0; i < CLIENT_COUNT; i++) {
    newSocketClient(i);
}

function newSocketClient(idx) {
    var socket = require('socket.io-client')('http://127.0.0.1:'+PORT, {forceNew: true});
    socket.on('connect', function(){
        console.log('connected', idx);
        socket.disconnect();
        console.log('disconnected', idx);
    });
}

[94cstyles]

In a production environment is not recommended use global.gc (),initially I was using global.gc () release the memory, but this method makes socket connection is unstable, resulting in dropped calls. Socket.io will take the initiative to release memory because the inert recycling and reuse connections, leading to see the memory has not been released.

[darrachequesne]

The memory is cleaned up by the garbage collectors periodically.

To me, memory is only** reclaimed if/when the OS needs it (if not triggered manually with global.gc()). So that doesn't seem like a leak. Or am I wrong?

** that's obviously more complex than that, but you get the picture 👼

[niranjanMeteor]

I am running 2 nodejs servers with socket io. both of them are proxied equally by nginx for load . so individual socket server is handling around 10K active connections . And process memory starts increasing from 300 MB initially to 3000MB finally within 2 days . I am using socket io 1.3.7 in production with node 4.4

[creole]

Similar issue here. I clean up unused sockets with:

sckt.removeAllListeners();
sckt.disconnect(true);

But memory consumption keeps growing. Development server with socket.io 1.4.8 and node 0.10.33.

2016-09-13 10:37 GMT+02:00 niranjanMeteor [email protected]:

I am running 2 nodejs servers with socket io. both of them are proxied
equally by nginx for load . so individual socket server is handling around
10K active connections . And process memory starts increasing from 300 MB
initially to 3000MB finally within 2 days . I am using socket io 1.3.7 in
production with node 4.4

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#2427 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF_1Xuz7Q-JCwjj4OvtWtTT1YLSsIlHsks5qpmDigaJpZM4HRQLl
.

[darrachequesne]

That issue was closed automatically. Please check if your issue is fixed with the latest release, and reopen if needed (with a fiddle reproducing the issue if possible).

[will-hang]

lol was this fixed? i'm scared to use sockets.io because i keep hearing about this issue

[eltharynd]

tbh it seems like the problem is still there... I'm trying to find a memory leak in my backend app but everything points to something inside of socket.io

[Fabbok1x]

This is not good. Issue still seems to exist. Definetly a very critical bug right here.

[bor8]

ping

[Novivy]

Hello?

in last version (4.1.3), disconnected sockets (from like network loss or just closing the browser) will remain in the memory forever...

You can easly check this with IO.sockets.sockets

In previous version we were able to loop over IO.sockets.connected and if the row was === to undefined, we could drop it. But now we only have IO.sockets.sockets, and all row are undefined, connected or not so that's not a solution anymore.

And relying on "disconnect" event is not a solution as it does not trigger 100% of the time a connection gets dropped

[MtDalPizzol]

Yup... the issue is still there and I found it in the worst scenario possible. A production server exploded.

It should be AT LEAST stated in the docs that the sockets objects are never deleted from memory and where they are stored, so we can come up with a way to get rid of them.

[SujithManjavana]

I was trying to load test my socket.io dev server. Initially, when I checked the memory usage with pm2 it was 44MB. It went up to 115MB during the test and remained 115MB even after I terminated the test program. "socket.io": "^4.5.0"

[MtDalPizzol]

For those who are about to lose their heads with this:

What solved the leak for me was this: force the clients to connect via the "websocket" protocol.

import { io } from 'socket.io-client'

// ...

io(url, {
   transports: ['websocket']
})

I believe this is because socket.io defaults to HTTP polling in some scenarios, even if the client supports websocket. I'm not sure, since I don't know the internals of the package and also if this is the expected behavior or a bug. What I DO KNOW is that I'm running a chat app for over a year now and everything is running smoothly without leaks since I took this route...

[SujithManjavana]

@MtDalPizzol my setup uses websockets only but still, it has this issue.

[thomaskleiven]

Also experiencing the same as @SujithManjavana when using "socket.io": "^4.5.3" and node v18.12.1

[darrachequesne]

Hi everyone, I wasn't able to reproduce the issue: https://socket.io/docs/v4/memory-usage/

Could you please check?

[Fabbok1x]

Ok, update on this. I have been able to fix the issue. However I don't have the exact fix available anymore. I might provide it in the future but I'll have to look up the code base. What I remember is there's an array/object which does never get cleared. I think it's the internal onclose that wouldn't remove a connection. It's definitely fixable I had it fixed back then.

[adamreisnz]

@Fabbok1x can you dig up the fix so others can also potentially apply it?

I am still running into issues where clients that suffer a local loss of internet connection, are never able to reconnect automatically again, even after I restart their machine they run on, unless I restart the Socket IO server. I believe it also has something to do with sockets not being properly removed after a passive disconnect of this nature.

[llaakso]

At least in 2.0.3 this kind of fix is needed:

io.on('connection', function (socket) {
  const intv = setInterval(() => {
    if (new Date().getTime() > socket.ping + 60000 * 10) {
      socket.disconnect();
      socket.removeAllListeners();
      socket = null;
      clearInterval(intv);
    }
    // maintain socket.ping by app
  }, 10000);
});

[darrachequesne]

@adamreisnz that sounds weird indeed. Do you have something that could help us dig into this? Some logs, dump of the client state? Which version of the Socket.IO client are you using?

@llaakso could you please explain your use case? There is already a heartbeat mechanism included in Socket.IO, which periodically checks the state of the connection.

Reference: https://socket.io/docs/v4/how-it-works/

Also, 2.0.3 is quite an old version, could you please check with more recent versions?

[llaakso]

@darrachequesne I'm aware that heartbeat mechanism exists, but it just do not seem to work as 10min dead connections still can be found. I don't know if my app has some logic which causes this, however, I don't think that should be possible and Socket.io should take care of it anyway as that is the core idea of the library, isn't it?

Due numerous undocumented breaking changes, we stuck to v2 and could not upgrade. Perhaps this is fixed in v4, but it seems there are people still struggling with this.

[darrachequesne]

@llaakso thanks for the honest feedback 👍

but it just do not seem to work as 10min dead connections still can be found

I understand, but the thing is, if we can't reproduce the issue, then fixing this will be quite hard.

Due numerous undocumented breaking changes

Do you have a specific example in mind? I'd really like to update the migration guide with the missing parts.

Reference: https://socket.io/docs/v4/migrating-from-2-x-to-3-0/

[RC0D3]

And I that try to ping with webbrowser in port of websocket , its work at /socket.io. But in / on this port crash my socket ultil I close my browser, and node js throw errors about memory leak.

WHY I CHOOSE THAT AND NOT LARAVEL WITH PHP SOCKETS WHY WHY WHY???? (Ratchet is not so good... but works very well, I prefer things that operate with laravel, better ;) )

[darrachequesne]

@RC0D3 could you please provide all necessary details, so we can help you?

[cody-evaluate]

seeing this issue in 4.7.5

[darrachequesne]

@cody-evaluate could you please provide all necessary details, so we can help you?