rpmalloc causes crash on exiting LMMS

[JohannesLorenz]

Bug Summary

On exiting LMMS, LMMS crashes with the following terminal output (see Logs for the backtrace):

lmms: /data/cprogs/lmms/master/src/3rdparty/rpmalloc/rpmalloc/rpmalloc/rpmalloc.c:1248: _rpmalloc_span_finalize: Assertion `span->list_size == span->used_count' failed.
Aborted (core dumped)

Caused by rpmalloc update in c58c781. Might be not our fault.

Steps to reproduce

Compile current master. Start LMMS. Close LMMS (File->Quit).

Expected behavior

No crash.

Actual behavior

Crash.

Affected LMMS versions

Master is affected since c58c781.
stable-1.2 is not affected.

Logs

Click to expand

(gdb) bt
#0  0x00007ffff60f2615 in raise () at /usr/lib/libc.so.6
#1  0x00007ffff60db862 in abort () at /usr/lib/libc.so.6
#2  0x00007ffff60db747 in _nl_load_domain.cold () at /usr/lib/libc.so.6
#3  0x00007ffff60eabf6 in  () at /usr/lib/libc.so.6
#4  0x0000555555933ffe in _rpmalloc_span_finalize (heap=0x7ffff3190058, iclass=66, span=0x7ffff2d80000, list_head=0x7ffff3190698) at /data/cprogs/lmms/master/src/3rdparty/rpmalloc/rpmalloc/rpmalloc/rpmalloc.c:1248
#5  0x0000555555935843 in _rpmalloc_heap_finalize (heap=0x7ffff3190058) at /data/cprogs/lmms/master/src/3rdparty/rpmalloc/rpmalloc/rpmalloc/rpmalloc.c:1833
#6  0x00005555559348b9 in _rpmalloc_heap_global_finalize (heap=0x7ffff3190058) at /data/cprogs/lmms/master/src/3rdparty/rpmalloc/rpmalloc/rpmalloc/rpmalloc.c:1430
#7  0x0000555555937b35 in rpmalloc_finalize () at /data/cprogs/lmms/master/src/3rdparty/rpmalloc/rpmalloc/rpmalloc/rpmalloc.c:2699
#8  0x00005555557a0e17 in MemoryManagerGlobalGuard::~MemoryManagerGlobalGuard() (this=0x555555b13070 , __in_chrg=) at /data/cprogs/lmms/master/src/core/MemoryManager.cpp:37
#9  0x00007ffff60f4db7 in __run_exit_handlers () at /usr/lib/libc.so.6
#10 0x00007ffff60f4f5e in  () at /usr/lib/libc.so.6
#11 0x00007ffff60dd159 in __libc_start_main () at /usr/lib/libc.so.6
#12 0x000055555571063e in _start ()

[JohannesLorenz]

I think the rpmalloc submodule could be bisected to find the issue, before we report it to the rpmalloc team.

[zonkmachine]

OK. I'm going to bisect it. I did that yesterday but I may not have gone far back in time.

[zonkmachine]

@JohannesLorenz This is with the -DCMAKE_BUILD_TYPE=Debug compile flag right?

[zonkmachine]

I needed to skip some parts of the bisect because I got compile errors so this is incomplete. I'll look into manually tweeking between the steps to see if I can get down to the last commit. In the meantime it's one of these:
mjansson/rpmalloc@3792374
mjansson/rpmalloc@c576814
mjansson/rpmalloc@68c6ae5

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
3792374b5aac7c86daaa3911179c64c56fe607fd
c576814c0d7ebb4c9c8303775b8565239ff25ac6
68c6ae5839ccb42b71e1c1f192d0f7603ae480d5
We cannot bisect more!

[zonkmachine]

The 'guilty' commit is mjansson/rpmalloc@68c6ae5

commit 68c6ae5839ccb42b71e1c1f192d0f7603ae480d5 (HEAD)
Author: Mattias Jansson <[email protected]>
Date:   Mon Jan 6 23:10:14 2020 +0100

    Refactor span handling (#133)

...
3rdparty/rpmalloc/rpmalloc/rpmalloc/rpmalloc.c:2158: _memory_span_finalize: Assertion `free_blocks == block_count' failed.
Aborted (core dumped)

The crash is in an assert that was added in this very same commit so I guess it could be that the issue was there already and that adding the assert makes lmms take a dive in some cases on a debug build.

//If this assert triggers you have memory leaks
	assert(free_blocks == block_count);

@JohannesLorenz Have you seen this crash on a non debug build?

[JohannesLorenz]

@JohannesLorenz Have you seen this crash on a non debug build?

Good observation. The error does not occur in a non-debug build.

[JohannesLorenz]

@zonkmachine Do you want to report the issue to rpmalloc, so they can find out if it's their bug or ours?

[zonkmachine]

@zonkmachine Do you want to report the issue to rpmalloc, so they can find out if it's their bug or ours?

No thank you good sir. I'm not on top of this one at all.

[JohannesLorenz]

Tracked in rpmalloc now as mjansson/rpmalloc#202.

[JohannesLorenz]

rpmalloc confirmed that it's most likely a leak in LMMS, so our bug. So this needs to be analyzed by us.

[mjansson]

@JohannesLorenz Have you seen this crash on a non debug build?

Good observation. The error does not occur in a non-debug build.

The reason for this is that I assume you only enable asserts in rpmalloc in debug builds?

As I mentioned in mjansson/rpmalloc#202 you could if you don't care about the leak (since the OS will release pages anyway on process termination) just disable asserts in rpmalloc until you can track this down.

[PhysSong]

The reason for this is that I assume you only enable asserts in rpmalloc in debug builds?

Yes.

it's most likely a leak in LMMS

I tried to track the origin by setting breakpoints at rpmalloc and rpfree. It seems like NotePlayHandleManager doesn't free s_available at exit.

[sziegler103]

Just a newbie developer here trying to provide some more insight on this issue.

The thing is, as far as I can tell this leak is occurring right before if not at program termination instead of in the middle of runtime.

Using the information from @PhysSong 's debug-fu, I made a little member function called free() for NotePlayHandleManager, calling MM_FREE for s_available and shoved it right before main.cpp's return statement

(NotePlayHandleManager.cpp)
\/---------------------------------------------------\/
(main.cpp)

which does seem to at least soothe the rpmalloc assert, though I'm not sure if it's solving the issue or just hiding it. I have no idea how it works. I'm not suggesting this as a fix, but I do hope it provides a little more information on what's going on.

[JohannesLorenz]

@PauseForAffliction This seems like a valid fix. I would have used std::unique_ptr<NotePlayHandle*>, but this code will be rewritten anyways when we will make LMMS realtime safe.

Can you please submit a pull request?

[sziegler103]

@JohannesLorenz Gotcha, submitted.

[zonkmachine]

Edit: Not related to this issue...
I just got this crash. I don't think I was closing lmms but I was probably loading a new project.

https://gist.github.com/zonkmachine/bcd7b134141ad17727ff9602514ca948

My rpmalloc is on:

commit 8d790d2b45e1818e531c61bf649c5225556dd07a (HEAD)
Author: Dominic Clark <[email protected]>
Date:   Sun Oct 4 12:07:48 2020 +0100

    Fix includes on MinGW-w64 (#200)

[PhysSong]

The backtrace says you've got a SIGFPE first, and then got a assertion failure while exit()ing.

[zonkmachine]

Ah, then it's most likely an underflow since it's the FxMixerView::updateFaders(). I've seen a whole lot of those. Thanks!