runtime: support dlclose with -buildmode=c-shared

[mattn]

package main

import (
    "C"
    "fmt"
)

var (
    c chan string
)

func init() {
    c = make(chan string)
    go func() {
        n := 1
        for {
            switch {
            case n%15 == 0:
                c <- "FizzBuzz"
            case n%3 == 0:
                c <- "Fizz"
            case n%5 == 0:
                c <- "Buzz"
            default:
                c <- fmt.Sprint(n)
            }
            n++
        }
    }()
}

//export fizzbuzz
func fizzbuzz() *C.char {
    return C.CString(<-c)
}

func main() {
}

build this with

$ go build -buildmode=c-shared -o libfizzbuzz.so libfizzbuzz.go

then go

from ctypes import *
import _ctypes
lib = CDLL("./libfizzbuzz.so")
lib.fizzbuzz.restype = c_char_p
print lib.fizzbuzz()
print lib.fizzbuzz()
print lib.fizzbuzz()
print lib.fizzbuzz()
print lib.fizzbuzz()
print lib.fizzbuzz()
_ctypes.dlclose(lib._handle)

1
2
Fizz
4
Buzz
Fizz
Segmentation fault

[minux]

I don't understand. What do you expect otherwise?
The code is still running, and you've unmapped its
pages.

In general, it's impossible to dlclose a Go shared
library (or a plugin).

[mattn]

eventhough close(c) and wait exiting goroutine, it reproduce.

On 6/7/15, Minux Ma [email protected] wrote:

I don't understand. What do you expect otherwise?
The code is still running, and you've unmapped its
pages.

---

Reply to this email directly or view it on GitHub:
#11100 (comment)

• Yasuhiro Matsumoto

[minux]

You can't stop the runtime, which uses its own OS threads.

[ianlancetaylor]

I think this is a legitimate feature request. Although we currently do not support calling dlclose on a Go shared library, and it would be difficult to make it work, it is not fundamentally impossible.

[minux]

It's possible if the user can be certain it doesn't hold onto any Go objects and all resources allocated by Go code has been freed (esp. no background goroutines). However, as there is no way to kill a goroutine, I think all sufficiently sophisticated Go shared library will not be unloadable. For example, the os/signal package contains a background goroutine to check for newly arrived signals. Any use of os/signal.Handle will leave the Go shared library un-unloadable. I'm sure there are other cases. What's more, the runtime can't reliably detect whether it's safe to unload the Go shared library, so rather than make dlclose potentially trigger segmentation fault later, I'd rather make dlclose always fail and document that.

[mattn]

This also reproduce.

package main

import (
        "C"
        "fmt"
)

var (
        c chan string
        q chan struct{}
)

func init() {
        c = make(chan string)
        q = make(chan struct{})
        go func() {
                defer func() {
                        recover()
                        q <- struct{}{}
                }()
                n := 1
                for {
                        switch {
                        case n%15 == 0:
                                c <- "FizzBuzz"
                        case n%3 == 0:
                                c <- "Fizz"
                        case n%5 == 0:
                                c <- "Buzz"
                        default:
                                c <- fmt.Sprint(n)
                                println("stop")
                        }
                        n++
                }
        }()
}

//export fizzbuzz
func fizzbuzz() *C.char {
        return C.CString(<-c)
}

//export finish
func finish() {
        close(c) // occur panic of sending closed channel in above
        <-q      // wait goroutine
}

func main() {
}

from ctypes import *
import _ctypes
lib = CDLL("./libfizzbuzz.so")
lib.fizzbuzz.restype = c_char_p
print lib.fizzbuzz()
print lib.fizzbuzz()
print lib.fizzbuzz()
print lib.fizzbuzz()
print lib.fizzbuzz()
print lib.fizzbuzz()
lib.finish()
_ctypes.dlclose(lib._handle)

[mattn]

i added runtime.LockOSThread() in top of init() but not.

[gcatlin]

I have a need for this functionality too. My goal is to be able to make changes to a running Go game engine without needing to completely reload the game.

The implementation idea is to have 2 layers, the platform layer and the game layer.

The platform layer is GOOS/GOARCH-specific and is written in C. It is responsible for the game loop wherein it gathers controller input and manages image and sound buffers that it provides to the game layer for writing. It uses OS-provided functionality to output graphics and sound.

The game layer is platform agnostic, contains game logic only, and is written in Go. It is called by the platform layer once per iteration of the game loop. The game layer takes the controller input and buffers provided by the platform layer, updates the game state, and writes to the buffers.

The game layer is a shared library, built using -buildmode=c-shared, that can be edited and recompiled at any time. When the platform layer detects that the shared library was modified, it unloads the previous version of the library (dlclose) then loads the new version of the library (dlopen).

See https://gist.github.com/gcatlin/e09359f6e53f37e74a82

I'm trying this on darwin/amd64 and getting the following error when calling dlopen on the shared library for the second time (i.e. after detecting that the shared library has changed):

runtime/cgo: could not obtain pthread_keys
    tried 0x101 0x102 0x103 0x104 0x105 0x106 0x107 0x108 0x109 0x10a 0x10b 0x10c 0x10d 0x10e 0x10f 0x110 0x111 0x112 0x113 0x115 0x116 0x117 0x118 0x119 0x11a 0x11b 0x11c 0x11d 0x11e 0x11f 0x120 0x121 0x122 0x123 0x124 0x125 0x126 0x127 0x128 0x129 0x12a 0x12b 0x12c 0x12d 0x12e 0x12f 0x130 0x131 0x132 0x133 0x134 0x135 0x136 0x137 0x138 0x139 0x13a 0x13b 0x13c 0x13d 0x13e 0x13f 0x140 0x141 0x142 0x143 0x144 0x145 0x146 0x147 0x148 0x149 0x14a 0x14b 0x14c 0x14d 0x14e 0x14f 0x150 0x151 0x152 0x153 0x154 0x155 0x156 0x157 0x158 0x159 0x15a 0x15b 0x15c 0x15d 0x15e 0x15f 0x160 0x161 0x162 0x163 0x164 0x165 0x166 0x167 0x168 0x169 0x16a 0x16b 0x16c 0x16d 0x16e 0x16f 0x170 0x171 0x172 0x173 0x174 0x175 0x176 0x177 0x178 0x179 0x17a 0x17b 0x17c 0x17d 0x17e 0x17f 0x180 0x181
fatal error: cgo callback before cgo call

Is there a different way to achieve this with Go?

[z505]

Minux said:

It's possible if the user can be certain it doesn't hold onto
any Go objects and all resources allocated by Go code
has been freed (esp. no background goroutines).

However, as there is no way to kill a goroutine, I think all
sufficiently sophisticated Go shared library will not be unloadable.

Hi, so how does go know it is safe for the exe/elf to close if a user kills the application or hits the close button on a win32 gui window... There must be some way of finding out if it is safe? or go program just exits badly with hanging data around?
Or how do you halt/exit a go app safely (an exe, not a dll)..? Would exiting a go app (exe) safely be the same as unloading a dll safely?

The go runtime does not keep a reference count of the number of goroutines currently open? This should not be a performance hit to keep track of N number of goroutines open? Each goroutine opened increments a ref count by 1. But maybe there is already something implemented like this, that could be tapped into and used for dll's to know if all goroutines are finished.

Minux says:

For example, the os/signal package contains a background
goroutine to check for newly arrived signals.

Can you increment a reference counter and unload the library later once all signals are finished (goroutine ref count hit zero)? How does the exe/elf normally exit safely and kill the program if goroutines could be still running? Simply make the DLL unload function act like how a normal go exe program safely exits? easier said than done, likely :-)

[glycerine]

Minux wrote,

You can't stop the runtime, which uses its own OS threads.

This is the root of the problem, @z505. I don't believe there is any graceful shutdown of the runtime threads under normal process exit/termination. Hence they may still have references to DLL memory. So currently there's no safety problem with process termination; just doing an exit() call, it simply unceremoniously stops the process, and doesn't need to do any graceful cleanup.

or go program just exits badly with hanging data around?

Yep.

[brutestack]

I'm using Go c-shared library in my Unity3D game.
Unity itself does not support unloading libraries, but when you try to exit Unity Editor or Unity based game it starts waiting untill all libraries complete their work (wait untill all threads stop)

In case of using Go c-shared library which creates at least one goroutine Unity will wait forever and will not be able to exit. This will not happen if library does not utilize goroutines.
So, go creates thead for my goroutine and never terminates it even if there is nothing to do in this thread.

Here is sample library code to reproduce:

package main

import "C"
import "time"

//export TestLib
func TestLib() {
	go func() {
		for i := 0; i < 10; i++ {
			fmt.Println("testlib.dll is here")
			time.Sleep(3 * time.Second)
		}
		fmt.Println("testlib.dll thread Done")
	}()
}

func main() {
}

I'm compiling that library on Linux that way:
env GOOS=windows GOARCH=amd64 CGO_ENABLED=1 CXX=x86_64-w64-mingw32-g++ CC=x86_64-w64-mingw32-gcc go build -o $(go env GOPATH)/lib/win64/testlib.dll -buildmode c-shared unity3d.com/testlib

Here is c# class that utilizes this library:

using System;
using System.Runtime.InteropServices;
using UnityEngine;

public class TestLibrary : IDisposable {
	public TestLibrary()
	{
		ServerProcess();
	}

	private void ServerProcess()
	{
		TestLib();
	}
		
	public void Dispose()
	{
	}


	[DllImport ("testlib")]
	private extern static void TestLib ();

}

Unity is besed on Mono, that's why c#.

There is a very-very dirty trick that fixes my problem and allows to exit my game without wating forever - I have implemented function in my Go library that causes panic and library crashes when I need to stop everything and exit game. Actually nobody should do that, but it works.
Additional code in Go library:

//export Panic
func Panic() {
	var panicChan chan bool
	close(panicChan)
}

Wortst of it is that I don't know what exactly crashes after calling Panic() from C# - Unity(Mono) or Go library, but I suspect Unity.
Anfortunately crash is the only way to exit Unity game that uses Go c-shared library with goroutines...

Another workaround is to compile Go code as executable and run it in separate process, but this is not sutible for iOS version of my game because iOS does not let executing separate processes (even included in the same application boundle) without using private APIs

[glycerine]

I'll attempt this. Background from https://groups.google.com/forum/#!topic/golang-nuts/L-tby34r5Gs

Ian wrote

Jason wrote> Where does the source for the runtime scheduler and garbage collector live these days?
The central locations are runtime/proc.go and runtime/mgc.go.
Jason wrote> Wherefore, I need to locate all runtime background threads and add in a means to shut them down upon request.
That's not currently supported, but it may be possible to modify the
scheduler to do it. There is no simple way.

Ian

[glycerine]

After some thinking, I realized that the main use case is really during development: to unload a Go DLL, and then re-load a modified version of that DLL; this being done during coding and evolving the DLL. We don't really want to stop the runtime, because we'll just then need to restart it upon the re-load of the newer version of the DLL.

So I propose the following, more general, approach to supporting dlclose with -buildmode=c-shared:

a) On windows, additionally during build of Go from source, build the runtime as a distinct DLL. Once the runtime DLL is loaded, it will never be unloaded, even if client Go DLLs that depend on the runtime are unloaded. This solves the tricky part of trying to halt the runtime cleanly, because we don't need to do that after all.

b) On windows, build Go DLLs as libraries that are clients of the Go runtime DLL. Each client Go DLL will dynamically load the Go runtime DLL if it is not already loaded, taking care prevent race on load by some means when there are two or more Go DLLs loaded during process start (I expect this to be a common race; perhaps the first to claim a pre-agreed upon localhost port wins the race and gets to load the runtime DLL from some well known location in GOROOT). Each Go DLL will increment the reference count on the runtime twice, so that Windows never unloads the runtime DLL, even if the client Go DLL is unloaded by a dlclose() call. A distinct new buildmode may be indicated, to distinguish it from c-shared which currently bundles another copy of the runtime into every DLL, and probably needs to continue to do so for backwards compatibility. Suggestion: buildmode=common-runtime-dll.

c) the extra benefit: now multiple DLLs when loaded all share the same runtime, and so the possibility of communicating via channels between DLLs becomes viable.

@ianlancetaylor @alexbrainman @minux and anyone else with wisdom to contribute: Feedback welcome.

[dchest]

@glycerine that may be the main use case for some, but others are writing shared libraries in Go to plug into other programs (e.g. see @brutestack's case above, or from my personal experience and @mattn's description for this issue, a library that is used from other programming languages) and run them in production, where they don't control how the library is used. I think your proposal, while useful for some cases, doesn't solve the issue.

[glycerine]

@dchest: This issue is about dlclose() support. The plan is perfectly compatible with plugging into non Go host programs. If you don’t understand why, ask about the specific point where you are confused, so that we can clarify. Also divide and conquer wise, we can’t do everything at once, so I’m okay with no runtime shutdown just prior to program termination. Especially as a workaround is already posted above.

[tmm1]

Looks like dlclose is currently ignored:

go/src/cmd/link/internal/ld/lib.go

Lines 1346 to 1348 in 70b1a45

	// Pass -z nodelete to mark the shared library as
	// non-closeable: a dlclose will do nothing.
	argv = append(argv, "-Wl,-z,nodelete")

It sounds like from this thread, an approximate solution would be to:

• revert above lines
• stop all user goroutines
• invoke panic from golang code and catch/ignore from C
• send SIGKILL to GC thread
• call dlcose()

Does that sound right?

[ianlancetaylor]

@tmm1 Not really. Consider #11100 (comment). Consider what should happen for a goroutine that is currently sitting in C code; if we remove the Go shared library then when the C code returns it will crash or (if some other shared library has been opened) behave unpredictably. Note that there is no separate GC thread; in the current Go runtime GC is handled by ordinary goroutines.

[tmm1]

I understand there are a lot of complexities and edge cases.

In my case I am working in a plugin environment where I have a lot of control and can ensure that no goroutines will be busy calling into C code or other shared libraries. I have the ability to make sure all my code is done before calling dlclose, so I'm wondering what's required/possible in that more limited scenario.

I suppose parts of the runtime internally could be calling into libc even if my user code is not. What I observe currently is that many of the OS threads related to the runtime start busy looping after I dlclose. I'll try to make a list of them and kill them to see if that helps.

[tmm1]

After looking more into this today, it seems there is no way to kill a thread. Even if you use pthread_kill (i.e tgkill), the signal will be delivered to that particular thread but the process as a whole will be affected by it if its a stop/terminate/kill signal.

What I observe currently is that many of the OS threads related to the runtime start busy looping after I dlclose.

I observed this behavior on macOS, where ps -M showed several threads at 100% cpu after dlclose. Using sample on the pid shows unknown backtraces, and lsof -nPp confirmed that the golang dylib was no longer mapped causing those runtime threads to be very confused.

It turns out -Wl,-z,nodelete does not work on macOS.

I found instead that using dlopen with RTLD_NODELETE has the same effect, and fixed the cpu usage issues I was having.

On Windows, a similar GetModuleHandleEx(GET_MODULE_HANDLE_EX_FLAG_PIN, ...) is required for the same effect.

My solution for now is to simply allow old copies of the runtime and my code to stay resident in memory. When my plugin updates, I load a new dylib/so with a fresh copy of the runtime and my code. These copies of the golang runtime seem to be happy residing side by side. The old runtimes are basically doing nothing anyway.

Note for anyone else attempting something similar: on macOS, be sure to pass RTLD_LOCAL (which is the default on Linux but not macOS). Without this flag, the handle returned by dlopen will be stale after renaming a dylib and moving a new one in its place. Similarly on Windows, LoadLibraryEx will return an old handle for a given full path to a dll, even if the dll has changed on disk. You must use a unique filename per version of the dll to be able to load multiple copies at the same time.

[iuridiniz]

This issue is about to get unloadable modules written in golang for some app that supports loadable modules.

During the development process, it's good do have this, in production, this could be useful if someone requires hot upgrades, but since that go runtime (threads) does not shutdown in dlclose, we don't have this.

I'm doing a hello world module example for freeswitch and I have this issue. The only thing I can do for now (golang <1.17) is to mark the module as unloadable.

Module: https://github.com/iuridiniz/freeswitch_module_golang_sample

[danieldonoghue]

@tmm1 Not really. Consider #11100 (comment). Consider what should happen for a goroutine that is currently sitting in C code; if we remove the Go shared library then when the C code returns it will crash or (if some other shared library has been opened) behave unpredictably. Note that there is no separate GC thread; in the current Go runtime GC is handled by ordinary goroutines.

@ianlancetaylor surely that's a problem for the c developer? it would be incumbent on them to stop the goroutine prior to closing the library; failing to do so would leave their application in an indeterminate state, as it would whatever resource they were trying to use after closing it?

[ianlancetaylor]

The problem is that there is nothing that the C developer can do to fix the problem. The Go runtime starts goroutines itself. Those goroutines can in some cases call into C code. There is no way for the program to stop those goroutines, or even to know whether they are running.

[iuridiniz]

@danieldonoghue
I think if golang provide something to shutdown itself (maybe runtime.Shutdown) will be nice, so we could export some function to be call this shutdown.

This simple shared library starts 6 threads in my machine (go 1.18) and I don't know to shutdown them correctly.

package main

import (
	"C"
)

func init() {
	// empty
}

//export answer
func answer() int8 {
	return 42
}

func main() {
}

calling it from python

import ctypes
import _ctypes
import time
import os


def get_answer_from_golang():
    lib = ctypes.CDLL("./libgoanswer.so")
    answer = lib.answer
    answer.restype = ctypes.c_int8

    print("Answer is", answer())

    _ctypes.dlclose(lib._handle)


def get_answer_from_c():
    lib = ctypes.CDLL("./libcanswer.so")
    answer = lib.answer
    answer.restype = ctypes.c_int8

    print("Answer is", answer())

    _ctypes.dlclose(lib._handle)


def get_number_of_threads():
    proc_status = "/proc/%d/status" % os.getpid()
    with open(proc_status) as f:
        v = f.read()
        return int(v.split("Threads:")[1].split()[0])


print("Number of threads is:", get_number_of_threads())
# Try commenting out the next line
get_answer_from_golang()
# get_answer_from_c()
while True:
    print("Number of threads is:", get_number_of_threads())
    time.sleep(1)

Results:

iuri@GTR:~/Sources/go/src/issue_11100$ go build -buildmode=c-shared -o libgoanswer.so
iuri@GTR:~/Sources/go/src/issue_11100$ python3 test_ctypes.py 
Number of threads is: 1
Answer is 42
Number of threads is: 7
Number of threads is: 7
^CTraceback (most recent call last):
  File "/home/iuri/Sources/go/src/issue_11100/test_ctypes.py", line 29, in <module>
    time.sleep(1)
KeyboardInterrupt

if I make the same library in C, there's no runtime after dlclose (what I think is the expecte behavior)

C version:

signed char answer()
{
    return 42;
}

Results:

iuri@GTR:~/Sources/go/src/issue_11100$ gcc -Wall -shared main.c -o libcanswer.so
iuri@GTR:~/Sources/go/src/issue_11100$ python3 test_ctypes.py 
Number of threads is: 1
Answer is 42
Number of threads is: 1
Number of threads is: 1
^CTraceback (most recent call last):
  File "/home/iuri/Sources/go/src/issue_11100/test_ctypes.py", line 40, in <module>
    time.sleep(1)
KeyboardInterrupt

[win-t]

I'm not an expert on how dynamic link works
but after reading man dlclose, we can provide a finalization function with __attribute__((destructor)) if the lib was written on C (I believe something like this can be achieved by golang linker)
so we can just make the finalization function hang and wait until the runtime is shut down properly

[gopherbot]

Change https://go.dev/cl/582975 mentions this issue: cmd/go: pass -Wl,-z,nodelete for c-shared mode with gccgo