Thread heaps aren't abandoned on MacOS

[colesbury]

The _mi_heap_done function first checks if the default heap is initialized. If it's not initialized, the function exits early without abandoning the thread's default heap.

mimalloc/src/init.c

Lines 232 to 234 in e946d56

	static bool _mi_heap_done(void) {
	mi_heap_t* heap = _mi_heap_default;
	if (!mi_heap_is_initialized(heap)) return true;

The problem is roughly that the thread-local variables have already been destroyed by the time _mi_heap_done is called. The access to _mi_heap_default resets it to _mi_heap_empty so the mi_heap_is_initialized check always returns false.

It's a little confusing because I think the __thread variables are handled by libdyld while the pthread-specific variables are managed by libpthread. Source code for both are at https://opensource.apple.com/release/macos-10145.html. The __thread variables (libdyld) are managed by pthread_key_t under the hood. The order of destruction is unspecified, but it appears that the __thread variables are destroyed before thread_done is called. (I'm not sure if this is deterministic).

Apple LLVM version 10.0.1 (clang-1001.0.46.4)
macOS Mojave 10.14.6
For simplicity I'm linking to the static libmalloc-debug.a and building with -DMI_OVERRIDE=OFF.

Test program:

https://gist.github.com/colesbury/1e3e2600e6ea5b71717fb9bfc4285d44

Instrument _mi_heap_done e.g.:

static bool _mi_heap_done(void) {
  mi_heap_t* heap = _mi_heap_default;
  if (!mi_heap_is_initialized(heap)) {
    printf("heap not initialized\n");
    return true;
  }

  printf("heap done\n"); // This doesn't get called!
  ...
}

[colesbury]

Here is an another example that leaks memory, in case it's easier to reproduce:

#include <thread>
#include <mimalloc.h>
#include <stdio.h>

void t1main() {
    void *p = mi_malloc(1024);
    if (!p) {
        printf("out of memory");
        abort();
    }
    mi_free(p);
}


int main() {
    while (1) {
        auto t1 = std::thread(t1main);
        t1.join();
    }
    return 0;
}

[colesbury]

One fix is to have mi_pthread_key always store the current value of _mi_heap_default. The mi_pthread_key destructor is called with this value, which avoids the need to re-access _mi_heap_default during thread destruction.

[daanx]

Hi Sam -- Awesome for finding this -- I cannot test it today but I will merge early next week! Thanks!