Fix memory leakage Tribler.dispersy.candidate.Candidate #442

synctext · 2015-07-09T08:27:37Z

we have a small memory leak.

4000 Tribler.dispersy.candidate.Candidate objects are in memory after 30h.
145313 Tribler.dispersy.candidate.Candidate objects are in memory after 784h.

Suspects for memory leakage. Our automatic reports with difficult 60MByte traces: http://jenkins.tribler.org/job/Test_performance_Tribler_idle_24H_nightly_devel/605/artifact/output/memprof/2180/

Elric has some tools parsing this. These can identify the possible dictionary.

Using Dowser we see the following suspects:
Tribler.dispersy.candidate.Candidate
Tribler.dispersy.candidate.WalkCandidate
Tribler.dispersy.member.Member
(probably linked to their key: Tribler.dispersy.crypto.M2CryptoPK)

Tribler shows 1108 channels, Dowser stores
Tribler.Main.Utility.GuiDBTuples.Channel

At 11:00am, after running for 24hours:

8 hours later:

NielsZeilemaker · 2015-07-09T10:50:44Z

How many communities did you have in memory?

synctext · 2015-07-09T11:28:58Z

standard testing. When doing 'mp3' search certains objects become big. They nicely disappear with a result list of just a few items. Same for channel browsing. That is all good.

These dispersy items never decrease. Slowly grow with each new discovered peers.

Standard Tribler default communities.

synctext · 2015-08-17T09:34:23Z

After running Tribler for 4 weeks 4 days and 16hours: thousands of objects in memory.

ps aux | grep tri
pouwelse 13289  0.0  0.0   4472     0 pts/20   S+   Jul15   0:00 /bin/sh ./tribler.sh
pouwelse 13293 29.1 18.4 5631764 1416800 pts/20 Sl+ Jul15 13711:40 python2.7 Tribler/Main/tribler.py

Only a few channels loaded:

NielsZeilemaker · 2015-08-19T19:46:31Z

Could you start the wx inspectiontool (ctrl+i), and then in the console

for community in dispersy.get_communities():
    print community.get_classification(), len(community.candidates)

synctext · 2015-12-01T13:52:59Z

2.2 GByte memory usage after running from 5Nov - 1Dec. Plus 231GByte of IO and 100h of CPU. (devel 785f442ec6fef5fffaa33bfbf086bd22bf401c87)

ps aux | grep tribler && date
pouwelse 26676 16.2 29.2 6479460 2252572 pts/23 Sl+ Nov05 6016:18 python2.7 Tribler/Main/tribler.py
Tue Dec  1 14:51:32 CET 2015

NielsZeilemaker · 2015-12-01T13:54:13Z

And the output of the command I posted above?

synctext · 2015-12-01T13:59:05Z

hey! @NielsZeilemaker
Can't get that WX inspection tool to start. Even tried understanding the documentation.. http://wiki.wxpython.org/Widget%20Inspection%20Tool

NielsZeilemaker · 2015-12-01T14:00:19Z

The key is only bound in the debug panel, first go there

synctext · 2015-12-01T14:05:21Z

Looks healthy.. Still missing 210000 instances...

NielsZeilemaker · 2015-12-01T14:12:43Z

And

for community in dispersy.get_communities():
    print community.get_classification(), len(community.request_cache._identifiers)

synctext · 2015-12-01T14:20:41Z

Hmmm..

NielsZeilemaker · 2015-12-01T14:28:33Z

And after

import gc
gc.collect()

synctext · 2015-12-01T14:29:06Z

checking..

synctext · 2015-12-01T14:30:59Z

gc.collect()
68

trying dowser again..

NielsZeilemaker · 2015-12-01T16:08:40Z

Let's remove the communities one by one.
Verify the memory usage afterwards

c = dispersy.get_communities()[0]
c.unload_community()

print "unloaded", c.get_classification()
print "remaining"

for community in dispersy.get_communities():
    print community.get_classification()

import gc
gc.collect()

Not sure how Tribler will respond 😄 And remove them one by one.

synctext · 2015-12-01T16:22:43Z

after the unload attempt:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/community.py", line 1069, in unload_community
    self._request_cache.clear()
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/requestcache.py", line 230, in clear
    assert isInIOThread(), "RequestCache must be used on the reactor's thread"
AssertionError: RequestCache must be used on the reactor's thread

NielsZeilemaker · 2015-12-02T09:09:35Z

from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
    c = dispersy.get_communities()[0]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload()

print "remaining"
for community in dispersy.get_communities():
    print community.get_classification()

import gc
gc.collect()

synctext · 2015-12-02T10:49:39Z

Thnx Niels! Sadly after 4 weeks Tribler ran out of memory to use. Starting dowser and WX inspection tool surely didn't help. Will try again.

*** Error in `python2.7': munmap_chunk(): invalid pointer: 0x00007fa5f32b5a20 ***
Aborted (core dumped)

synctext · 2015-12-10T16:25:51Z

Python 2.7.10 on linux2, wxPython 2.8.12.1 (gtk2-unicode)
NOTE: The 'obj' variable refers to the object selected in the tree.
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
    c = dispersy.get_communities()[0]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload()

print "remaining"
for community in dispersy.get_communities():
    print community.get_classification()

import gc
gc.collect()

 print "remaining"
  File "<input>", line 1
    print "remaining"
    ^
IndentationError: unexpected indent
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
    c = dispersy.get_communities()[0]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload()
unloaded ChannelCommunity
print "remaining"
remaining
for community in dispersy.get_communities():
    print community.get_classification()

HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
SearchCommunity
DiscoveryCommunity
AllChannelCommunity
import gc
gc.collect()
76584

Memory was growing a bit, even after 75 of object where removed

XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   868 pts/23   S+   18:50   0:00 /bin/sh ./tribler.sh
pouwelse 17654 32.7  3.6 2056020 279700 pts/23 Sl+  18:50   0:15 python2.7 Tribler/Main/tribler.py
pouwelse 17687  0.0  0.0   9496  2140 pts/1    S+   18:51   0:00 grep tribler
Fri Dec  4 18:51:22 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   672 pts/23   S+   Dec04   0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.0  7.4 3729632 570700 pts/23 Sl+  Dec04 604:01 python2.7 Tribler/Main/tribler.py
pouwelse 26722  0.0  0.0   9500  2092 pts/1    R+   13:52   0:00 grep tribler
Tue Dec  8 13:52:56 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   296 pts/23   S+   Dec04   0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4  9.5 3836896 737932 pts/23 Sl+  Dec04 980:07 python2.7 Tribler/Main/tribler.py
pouwelse 29688  0.0  0.0   9500  2148 pts/1    S+   17:03   0:00 grep tribler
Thu Dec 10 17:03:49 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   296 pts/23   S+   Dec04   0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4 10.2 3849048 784964 pts/23 Sl+  Dec04 981:30 python2.7 Tribler/Main/tribler.py
pouwelse 30878  0.0  0.0   9500  1996 pts/1    S+   17:13   0:00 grep tribler
Thu Dec 10 17:13:14 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   296 pts/23   S+   Dec04   0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4 10.2 3849048 788756 pts/23 Sl+  Dec04 981:52 python2.7 Tribler/Main/tribler.py
pouwelse 30914  0.0  0.0   9500  2184 pts/1    S+   17:15   0:00 grep tribler
Thu Dec 10 17:15:17 CET 2015
XPS13:~/GITHUB/tribler>ps aux | grep tribler && date
pouwelse 17650  0.0  0.0   4476   296 pts/23   S+   Dec04   0:00 /bin/sh ./tribler.sh
pouwelse 17654 11.4 10.4 3849048 800956 pts/23 Sl+  Dec04 982:36 python2.7 Tribler/Main/tribler.py
pouwelse 31020  0.0  0.0   9500  2084 pts/1    S+   17:19   0:00 grep tribler
Thu Dec 10 17:19:41 CET 2015

After 1 unload:

Tribler.dispersy.candidate.Candidate
Min: 33606 Cur: 33607 Max: 33609 TRACE
Tribler.dispersy.candidate.WalkCandidate
Min: 25785 Cur: 25786 Max: 25788 TRACE

synctext · 2015-12-10T16:54:19Z

@NielsZeilemaker
Here you can see my embarrassing fiddling.
It does not free any candidate.Candidate, status after multiple unload() attempts:

Tribler.dispersy.candidate.Candidate
Min: 33612 Cur: 33692 Max: 33693 TRACE
Tribler.dispersy.candidate.WalkCandidate
Min: 25779 Cur: 26109 Max: 26122 TRACE

Console logfile:

Python 2.7.10 on linux2, wxPython 2.8.12.1 (gtk2-unicode)
NOTE: The 'obj' variable refers to the object selected in the tree.
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
    c = dispersy.get_communities()[0]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload()

print "remaining"
for community in dispersy.get_communities():
    print community.get_classification()

import gc
gc.collect()
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload():
    c = dispersy.get_communities()[0]
    print "unloaded", c.get_classification()

do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
do_unload()
unloaded ChannelCommunity
@blocking_call_on_reactor_thread
def do_unload2():
    c = dispersy.get_communities()[1]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload2()
unloaded HiddenTunnelCommunity
gc.collect()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
NameError: name 'gc' is not defined
import gc
gc.collect()
236
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload3(int num):
  File "<input>", line 2
    def do_unload3(int num):
                         ^
SyntaxError: invalid syntax
def do_unload3(num):
    c = dispersy.get_communities()[num]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload3(2)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "<input>", line 3, in do_unload3
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/community.py", line 1069, in unload_community
    self._request_cache.clear()
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/requestcache.py", line 230, in clear
    assert isInIOThread(), "RequestCache must be used on the reactor's thread"
AssertionError: RequestCache must be used on the reactor's thread
from Tribler.dispersy.util import blocking_call_on_reactor_thread
@blocking_call_on_reactor_thread
def do_unload3(num):
    c = dispersy.get_communities()[num]
    c.unload_community()
    print "unloaded", c.get_classification()

do_unload3(2)
unloaded ChannelCommunity
gc.collect()
12
do_unload(3)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/util.py", line 47, in helper
    return blockingCallFromThread(reactor, func, *args, **kargs)
  File "/home/pouwelse/GITHUB/tribler/Tribler/dispersy/util.py", line 238, in blockingCallFromThread
    result.raiseException()
  File "<string>", line 2, in raiseException
TypeError: do_unload() takes no arguments (1 given)
do_unload3(3)
unloaded PreviewChannelCommunity
gc.collect()
156
do_unload3(4)
unloaded ChannelCommunity
gc.collect()
63
do_unload3(5)
unloaded ChannelCommunity
do_unload3(5)
unloaded ChannelCommunity
do_unload3(6)
unloaded PreviewChannelCommunity
do_unload3(7)
unloaded PreviewChannelCommunity
for community in dispersy.get_communities():
    print community.get_classification()

PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
SearchCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
AllChannelCommunity
for community in dispersy.get_communities():
    print community.get_classification()

PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
AllChannelCommunity
do_unload3(7)
unloaded SearchCommunity
do_unload3(7)
unloaded SearchCommunity
for community in dispersy.get_communities():
    print community.get_classification()

PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
ChannelCommunity
ChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
AllChannelCommunity
do_unload3(3)
unloaded HiddenTunnelCommunity

do_unload3(5)
unloaded ChannelCommunity
do_unload3(6)
unloaded PreviewChannelCommunity
gc.collect()
183
do_unload3(7)
unloaded PreviewChannelCommunity
do_unload3(8)
unloaded ChannelCommunity
do_unload3(9)
unloaded PreviewChannelCommunity
do_unload3(10)
unloaded PreviewChannelCommunity
do_unload3(11)
unloaded SearchCommunity
do_unload3(0)
unloaded PreviewChannelCommunity
do_unload3(1)
unloaded PreviewChannelCommunity
do_unload3(2)
unloaded PreviewChannelCommunity
do_unload3(4)
unloaded PreviewChannelCommunity
do_unload3(3)
unloaded PreviewChannelCommunity
do_unload3(5)
unloaded PreviewChannelCommunity
do_unload3(6)
unloaded PreviewChannelCommunity
do_unload3(7)
unloaded SearchCommunity
do_unload3(8)
unloaded DiscoveryCommunity
do_unload3(9)
unloaded PreviewChannelCommunity
do_unload3(10)
unloaded PreviewChannelCommunity
do_unload3(11)
unloaded PreviewChannelCommunity
for community in dispersy.get_communities():
    print community.get_classification()

PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
AllChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
for community in dispersy.get_communities():
    print community.get_classification()

PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
HiddenTunnelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
ChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
SearchCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
DiscoveryCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
AllChannelCommunity
PreviewChannelCommunity
PreviewChannelCommunity
do_unload3(49)
unloaded PreviewChannelCommunity
do_unload3(50)
unloaded AllChannelCommunity
gc.collect()
80

NielsZeilemaker · 2015-12-11T14:24:29Z

So then probably something else is keeping a reference to the candidates. So that would be Tribler. Is there a debug panel which has any lists of candidates?

synctext added the bug label Jul 9, 2015

synctext mentioned this issue Jul 9, 2015

Update Dispersy pointer when the memory leak issue is closed/fixed Tribler/tribler#1551

Closed

synctext assigned devos50 Dec 1, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix memory leakage Tribler.dispersy.candidate.Candidate #442

Fix memory leakage Tribler.dispersy.candidate.Candidate #442

synctext commented Jul 9, 2015

NielsZeilemaker commented Jul 9, 2015

synctext commented Jul 9, 2015

synctext commented Aug 17, 2015

NielsZeilemaker commented Aug 19, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 1, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 1, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 1, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 1, 2015

synctext commented Dec 1, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 1, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 2, 2015

synctext commented Dec 2, 2015

synctext commented Dec 10, 2015

synctext commented Dec 10, 2015

NielsZeilemaker commented Dec 11, 2015

Fix memory leakage Tribler.dispersy.candidate.Candidate #442

Fix memory leakage Tribler.dispersy.candidate.Candidate #442

Comments

synctext commented Jul 9, 2015

NielsZeilemaker commented Jul 9, 2015

synctext commented Jul 9, 2015

synctext commented Aug 17, 2015

NielsZeilemaker commented Aug 19, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 1, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 1, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 1, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 1, 2015

synctext commented Dec 1, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 1, 2015

synctext commented Dec 1, 2015

NielsZeilemaker commented Dec 2, 2015

synctext commented Dec 2, 2015

synctext commented Dec 10, 2015

synctext commented Dec 10, 2015

NielsZeilemaker commented Dec 11, 2015