Opening a side module larger than 4KB fails on Chrome #11753

cuinjune · 2020-07-29T09:51:16Z

Hi, I'm trying to dynamically load a side module from the main module using dlopen().
The side module loads fine as long as its size is smaller than 4KB but I need to load large-size side modules.
Here's a simple code you can test this:

side.c:

#define SIZE 4000
char dummy[SIZE] = {};

int side(int a)
{
    return SIZE;
}

main.c:

#include <stdio.h>
#include <dlfcn.h>

int main() 
{
    void *handle;
    typedef int (*func_t)(int);

    handle = dlopen("side.wasm", RTLD_NOW);

    if (!handle) {
        printf("failed to open the library\n");
        return 0;
    }
    func_t func = (func_t)dlsym(handle, "side");

    if (!func) {
        printf("failed to find the method\n");
        dlclose(handle);
        return 0;
    }
    printf("side module size: %d byte\n", func(1));
}

index.html:

<!DOCTYPE html>
<html>

<head>
</head>

<body>
  <script async src="main.js"></script>
</body>

</html>

commands:

emcc side.c -s SIDE_MODULE=1 -o side.wasm
emcc main.c -s MAIN_MODULE=1 -o main.html --preload-file side.wasm
python3 -m http.server 8080

And this is the result I get in the Chrome browser:

Error in loading dynamic library side.wasm: RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.

Can someone please guild me on how to dynamically load a side module larger than 4KB?
Thank you in advance!

The text was updated successfully, but these errors were encountered:

gerald-dotcom · 2020-07-29T18:54:40Z

It's a Chrome limitation, you can use loadDynamicLibrary(url, {loadAsync: true, global: true, nodelete: true}) but there is one thing that I'm missing to make it work. Maybe someone can help you out.

kripken · 2020-07-29T21:27:30Z

It looks like the preloading logic looks for the normal shared library suffix on POSIX, .so,

emscripten/src/library_browser.js

Line 241 in df56ba1

return !Module.noWasmDecoding && name.endsWith('.so');

We should document that if it isn't already.

cuinjune · 2020-07-30T07:39:15Z

@sbc100 Could you please help me? I would really appreciate it!

sbc100 · 2020-07-30T18:47:23Z

Can you try using the .so extension as @kripken suggested? It looks like there is some special handling to preload side modules but it looks for the .so extension.

BTW in this mode you are going to delay the startup of you program until both the side module and main module are instantiated, so you are not really getting any benefit at runtime from using the a side module. Would it make more sense to simply compile the side module into the main program?

cuinjune · 2020-07-31T01:24:39Z

@sbc100 How do I build a side module as the so extension?
Can I just replace wasm with so like the following?

emcc side.c -s SIDE_MODULE=1 -o side.so

I need the side module to exist separately from the main module, and the side module will be instantiated in the middle of the program not necessarily at the startup.

sbc100 · 2020-07-31T01:32:49Z

Yes -o side.so should work. Better still. -o libside.so to match the UNIX convension.

If you preload the file with --preload-file then the side module will actually be instantiated at startup.. even though logically you don't access it until later. It should be be observable to you program other than the that the startup will be delayed until the side module has been downloaded and compiled. Does the work for you?

cuinjune · 2020-07-31T01:40:09Z

Yes, It's okay if there's a delay. I'm using the --preload-file for testing purposes now, but the side modules should be dynamically added to the file system and can be loaded in the middle of the program. That's why I'm using the side modules. And I already tested with the side modules smaller than 4KB and they work fine.

I used these commands to build:

emcc side.c -s SIDE_MODULE=1 -c -o side.so
emcc main.c -s MAIN_MODULE=1 -o main.html --preload-file side.so
python3 -m http.server 8080

Now I got these errors when I run it in Chrome:

Assertion failed: need the dylink section to be first

Error in loading dynamic library side.so: RuntimeError: abort(Assertion failed: need the dylink section to be first) at Error
at jsStackTrace (main.js:2568)
at stackTrace (main.js:2586)
at abort (main.js:2288)
at assert (main.js:1329)
at loadWebAssemblyModule (main.js:756)
at createLibModule (main.js:650)
at getLibModule (main.js:669)
at loadDynamicLibrary (main.js:708)
at _dlopen (main.js:7686)
at __original_main (:8080/:wasm-function[153]:0x6c7e7)

cuinjune · 2020-07-31T02:08:15Z

If loading the large-size side module with dlopen() is not possible, is there any workaround to this? (e.g. loading the .wasm file from index.html?)

sbc100 · 2020-07-31T02:45:27Z

Did you remember to change the name of side module in main.c?

Its should work with reloading such that side.so is loaded at start and you should not see a runtime call to . loadWebAssemblyModule on dlopen becasue side.so is already included in Module['preloadedWasm']. See

emscripten/src/library_browser.js

Line 252 in 8bd9c93

Module['preloadedWasm'][name] = module;

If that doesn't work you can try using the asynrous JS function loadDynamicLibrary as suggested earlier in this thread.
However the dlopen+preload approach should work if you want it to.

cuinjune · 2020-07-31T03:14:19Z

@sbc100 Yes I did of course.
Here's what everything looks like:

side.c:

#define SIZE 3000
char dummy[SIZE] = {};

int side(int a)
{
    return SIZE;
}

main.c:

#include <stdio.h>
#include <dlfcn.h>

int main()
{
    void *handle;
    typedef int (*func_t)(int);

    handle = dlopen("side.so", RTLD_NOW);

    if (!handle) {
        printf("failed to open the library\n");
        return 0;
    }
    func_t func = (func_t)dlsym(handle, "side");

    if (!func) {
        printf("failed to find the method\n");
        dlclose(handle);
        return 0;
    }
    printf("side module size: %d byte\n", func(1));
}

index.html:

<!DOCTYPE html>
<html>

<head>
</head>

<body>
  <script async src="main.js"></script>
</body>

</html>

commands:

emcc side.c -s SIDE_MODULE=1 -c -o side.so
emcc main.c -s MAIN_MODULE=1 -o main.html --preload-file side.so
python3 -m http.server 8080

Result:

Assertion failed: need the dylink section to be first

Error in loading dynamic library side.so: RuntimeError: abort(Assertion failed: need the dylink section to be first) at Error
at jsStackTrace (main.js:2568)
at stackTrace (main.js:2586)
at abort (main.js:2288)
at assert (main.js:1329)
at loadWebAssemblyModule (main.js:756)
at createLibModule (main.js:650)
at getLibModule (main.js:669)
at loadDynamicLibrary (main.js:708)
at _dlopen (main.js:7686)
at __original_main (:8080/:wasm-function[153]:0x6c7e7)

cuinjune · 2020-07-31T03:28:50Z

I tried to use the loadDynamicLibrary:

<!DOCTYPE html>
<html>

<head>
</head>

<body>
  <script>
	loadDynamicLibrary("side.wasm", {loadAsync: true, global: true, nodelete: true})
  </script>
  <script async src="main.js"></script>
</body>

</html>

But I get this error:

Uncaught ReferenceError: loadDynamicLibrary is not defined

gerald-dotcom · 2020-07-31T08:36:49Z

I tried to use the loadDynamicLibrary:
<!DOCTYPE html>
<html>

<head>
</head>

<body>
  <script>
	loadDynamicLibrary("side.wasm", {loadAsync: true, global: true, nodelete: true})
  </script>
  <script async src="main.js"></script>
</body>

</html>
But I get this error:

Uncaught ReferenceError: loadDynamicLibrary is not defined

Use EM_JS and rename .wasm to .so

cuinjune · 2020-07-31T23:30:59Z

@gerald-dotcom I still get the same error:

Uncaught (in promise) RuntimeError: abort(Assertion failed: need the dylink section to be first) at Error
at jsStackTrace (http://localhost:8080/main.js:2568:17)
at stackTrace (http://localhost:8080/main.js:2586:16)
at abort (http://localhost:8080/main.js:2288:44)
at assert (http://localhost:8080/main.js:1329:5)
at loadWebAssemblyModule (http://localhost:8080/main.js:756:3)
at createLibModule (http://localhost:8080/main.js:650:12)
at http://localhost:8080/main.js:665:16
at abort (http://localhost:8080/main.js:2294:11)
at assert (http://localhost:8080/main.js:1329:5)
at loadWebAssemblyModule (http://localhost:8080/main.js:756:3)
at createLibModule (http://localhost:8080/main.js:650:12)
at http://localhost:8080/main.js:665:16

Here's my code:

main.c:

#include <stdio.h>
#include <dlfcn.h>
#include <emscripten.h>

EM_JS(void, loadLibrary, (), {
      loadDynamicLibrary("side.so", {loadAsync: true, global: true, nodelete: true});
});

int main()
{
    loadLibrary();
}

commands:

emcc side.c -s SIDE_MODULE=1 -c -o side.so
emcc main.c -s MAIN_MODULE=1 -o main.html --preload-file side.so
python3 -m http.server 8080

sbc100 · 2020-08-01T00:03:41Z

Strange... somehow the side module is missing the dylink section which emscripten adds to the beginning of it.

cuinjune · 2020-08-01T00:23:41Z

I also posted my question on Stackoverflow https://stackoverflow.com/q/63150966/5224286 with a bounty.

sbc100 · 2020-08-01T01:08:48Z

Ah! The problem is that you are passing -c in emcc side.c -s SIDE_MODULE=1 -c -o libside.so which means "compile to an object file". This means the resulting side.so is not a shared libray but just an object file.

If you remove -c it should work.

Sadly it looks like you also have to change the output name back to .wasm to persuade emcc to produce a side module. We should fix that part.

cuinjune · 2020-08-01T01:23:10Z

@sbc100 But if I remove -c then it doesn't compile.
command:
emcc side.c -s SIDE_MODULE=1 -o side.so

I get the following error:

emcc: error: SIDE_MODULE must only be used when compiling to an executable shared library, and not when emitting an object file. That is, you should be emitting a .wasm file (for wasm) or a .js file (for asm.js). Note that when compiling to a typical native suffix for a shared library (.so, .dylib, .dll; which many build systems do) then Emscripten emits an object file, which you should then compile to .wasm or .js with SIDE_MODULE.

sbc100 · 2020-08-01T01:59:01Z

Sadly it looks like you also have to change the output name back to .wasm to persuade emcc to produce a side module. You will then need to rename the file .so when to preload it. We should fix that part.

sbc100 · 2020-08-01T02:01:42Z

Apologies for that very hard to understand error message... the situation with linking .so and .dylib extensions is complicated due to the history emscripten and keeping compataiblity with build systems that want to use those extensions but don't want side modules.

cuinjune · 2020-08-01T03:57:04Z

So is it not possible to load a side module larger than 4KB at the moment?

sbc100 · 2020-08-01T04:51:19Z

No, its possible, you just need to do the instantiation asynchronously because the browser doesn't support the synchronous loading of larger modules. This limitation is built into the browser and not related to emscripten.

Both loadDynamicLibrary (with loadAsync: true) and dlopen (in combination with the emscripten feature that pre-loads .so files) should allow for this asynchronous loading.

cuinjune · 2020-08-01T04:58:20Z

Thank you. so how can I generate the .so file?
Should I emcc side.c -s SIDE_MODULE=1 -o side.wasm and then rename the side.wasm to side.so?

sbc100 · 2020-08-01T05:01:22Z

Yes, if you want to use dlopen in combination with the preload plugin system that seems necessary today.

I will work two changes:

Allow linking side modules directly to .so.
Allow .wasm files to be preloaded by the preload plugin.

But you should be able to get it working even without either of those changes.

cuinjune · 2020-08-01T05:17:36Z

Thanks, I tried it as you said. Here's my code:

main.cpp

#include <stdio.h>
#include <emscripten.h>
#include <emscripten/bind.h>

using namespace emscripten;

int side(int a);

void hello() {
    printf("side module size: %d byte\n", side(1));
}

EMSCRIPTEN_BINDINGS(my_module) {
    function("hello", &hello);
}

EM_JS(void, loadLibrary, (), {
      loadDynamicLibrary("side.so", {loadAsync: true, global: true, nodelete: true});
});

int main()
{
    loadLibrary();
}

index.html

<!DOCTYPE html>
<html>

<head>
</head>

<body>
  <button id="buttonPressed">call side() function</button>
  <script>
    var Module
      = {
      preRun: []
      , postRun: []
      , print: function (e) {
        1 < arguments.length && (e = Array.prototype.slice.call(arguments).join(" "));
        console.log(e);
      }
      , printErr: function (e) {
        1 < arguments.length && (e = Array.prototype.slice.call(arguments).join(" "));
        console.error(e)
      }
    };

    function buttonPressed() {
      Module.hello();
    }

    window.onload = async () => {
      document.getElementById("buttonPressed").addEventListener("click", buttonPressed, false);
    };
  </script>
  <script async src="main.js"></script>
</body>

</html>

commands:

emcc side.c -s SIDE_MODULE=1 -o side.wasm
mv -f side.wasm side.so
emcc --bind main.cpp -s MAIN_MODULE=1 -o main.html --preload-file side.so
python3 -m http.server 8080

After the page is loaded, if I press the call side() function button, I get the following error:

main.js:2294 Uncaught RuntimeError: abort(external function '_Z4sidei' is missing. perhaps a side module was not linked in? if this function was expected to arrive from a system library, try to build the MAIN_MODULE with EMCC_FORCE_STDLIBS=1 in the environment) at Error

What did I do wrong?

sbc100 · 2020-08-01T06:05:50Z

That looks like a name mangling issue. If you compile your side module as C rather than C++ then you need to also add extern "C" to the declaration in your C++ code (main.cpp):

extern "C" {
  int side(int a);
}

sbc100 · 2020-08-01T06:08:37Z

While working on some fixed in this area I noticed that the preload+dlopen approach has some other issues so its good that proceed with the loadDynamicLibrary approach for now until we can address them.

cuinjune · 2020-08-01T06:23:35Z

It worked! Thank you so much!

May I ask how to implement a callback function to detect if the .so file is successfully loaded?

sbc100 · 2020-08-01T06:28:59Z

Looks like you can wait on the returned promise:

// - if flags.loadAsync=true, the loading is performed asynchronously and        
//   loadDynamicLibrary returns corresponding promise.

cuinjune · 2020-08-01T07:19:29Z

Thank you so much!
Is it not possible to dynamically call a side module's function (side(1);) without declaring it (int side(int a);) in the main module? This is the reason why I was using dlopen() so I can dynamically call a function with string.

sbc100 · 2020-08-01T15:39:51Z

Try using dlsym with RTLD_DEFAULT is the first argument. This will search global namespace of all symbols and should contains your side symbol after the code has been loaded.

AndyC-NN · 2022-02-20T04:12:15Z

The easy way to actually do this is to load the module with WebAssembly.instantiate in the javascript side. Use this tutorial to get beyond the 4K limit, https://developers.google.com/web/updates/2018/04/loading-wasm Once the module is loaded, get the function this way:

side = instance.exports._Z4sidev;

So we now have a javascript function of side but this now seems unusable to use in C++, until you realise that Emscripten has a handy function to get the function pointers for C/C++ called addFunction:

sidePtr = addFunction(side, 'vi');

The 'vi' to addFunction is really not that important here. It can be anything. That passes back an int that is a function pointer. Some of you will now wonder how can the int be a pointer? Under the hood of Emscripten compiling all functions in C/C++ after the code is compiled in WASM will be an int. With that knowledge you can just do this in C/C++:

typedef int (*side_function)();
side_function side = (side_function)sidePtr;
int size = side();

And somehow compiling with Emscripten this all works great!

aisnote · 2022-09-02T03:17:27Z

@cuinjune did you meet this kind of issue before?

Seems it is caused by my.wasm building settings?

EM_JS(void, doLoadLibrary, (), {
    Asyncify.handleAsync(async() => {
        try
        {
            await loadDynamicLibrary('ff/lib/my.wasm', {loadAsync : true, global : true, nodelete : true, fs : FS});
            //console.log('side module size: ' + Module['_side']());
        }
        catch (error)
        {
            console.log(error);
        }
    });
});

pd-l2ork · 2023-07-31T14:23:19Z

Following-up on this thread. @kripken I am working on the code base that @cuinjune originally developed and whose critical component is the solution developed in this thread. It appears that building that code on an older version of emscripten allows for loadDynamicLibrary to work. On the latest emscripten version it fails, as follows (please bear with me since I am new to emscripten and am still trying to find my way around, so apologies in advance if I am poorly describing the problem):

Whether you build in old or new version, it appears all the objects/plugins that may be loaded later when the programming language runs are "packaged" into the main.js file. By investigating this huge file, generated by emscripten, I find bunch of statements like the one below:

{"filename":"/pd-l2ork/extra/maxlib/average.wasm","start":15274628,"end":15276992}

The fiilename reflects the file's original location on the disk. I may be wrong about this, but it seems to me the last two numbers perhaps reflect where it is packed address-wise in the bundled file created by the compile/linking? Curiously, the older version had also , "audio": 0/1} appended at the end, suggesting the older version also offered differentiation between audio files and objects. Not sure why this is not there when compiled with a newer version, and whether that may be causing the aforesaid problem.

When running the demo code snippet on an older version of emscripten, additional objects open without the problem. On the latest version, the same request, instead of looking through its bundled objects, it issues a GET request looking for it inside the public/ folder (I suppose it is trying to wget it). The error on the console is as follows (removed calls before it that are irrelevant--it starts with the custom function call __PD_loadlib whose structure is effectively identical to the example @cuinjune posted above in Opening a side module larger than 4KB fails on Chrome #11753 (comment):

GET http://localhost:3000/emscripten/pd-l2ork-web/extra/maxlib/average.wasm 404 (Not Found)
readAsync	@	main.js:1
asyncLoad	@	main.js:1
(anonymous)	@	main.js:1
loadLibData	@	main.js:1
getExports	@	main.js:1
loadDynamicLibrary	@	main.js:1
(anonymous)	@	main.js:1
(anonymous)	@	main.js:1
handleSleep	@	main.js:1
handleAsync	@	main.js:1
__Pd_loadLib	@	main.js:1

Is the GET being called because the code is somehow not finding the object in question inside the bundle?

Copying precompiled wasm objects contained inside pd-l2ork-web inside the public/emscripten folder allows the object to be found via GET, but then opening of the object fails with an error where the main instantiation function symbol is not found. Namely:

error: load_object: Symbol "average_setup" not found

All objects have their objectname_setup as their entry point function and have been compiled as follows (please pardon the redundancy of the makefile flags--this is a huge code base with over a thousand of objects developed by 3rd parties, so the automated build process is a bit messy):

emcc -s SIDE_MODULE=1 -O3 -sEMULATE_FUNCTION_POINTER_CASTS=1 -o "average.wasm" "average.o"   
chmod a-x "average.wasm"

Below is also the relevant code that calls loadDynamicLibrary:

EM_JS(int, __Pd_loadLib, (const char *filename, const char *symname), {
    console.log("Pd-L2Ork __Pd_loadlib filename=<" + UTF8ToString(filename) + "> symname=<" + UTF8ToString(symname) + ">");
    return Asyncify.handleAsync(async () => {
        try {
            await loadDynamicLibrary(UTF8ToString(filename), {loadAsync: true, global: true, nodelete: true, fs: FS});
            var makeout = Module['_' + UTF8ToString(symname)];
            if (typeof makeout === "function") {
                makeout();
                console.log("...success");
                return 1; // success
            }
            else {
                console.log("...no function found");
                return -1; // couldn't find the function
            }
        }
        catch (error) {
            console.log("...failed");
            console.error(error);
            return 0; // couldn't load the external
        }
    });
});

The relevant build script code that bundles everything together on the emscripten side of things is as follows:

-s ASYNCIFY -s "ASYNCIFY_IMPORTS=['__Pd_loadLib']" \
-s USE_SDL=2 -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s ALLOW_MEMORY_GROWTH=1 \
-s FORCE_FILESYSTEM=1 -s "EXTRA_EXPORTED_RUNTIME_METHODS=['FS']" \
--no-heap-copy --preload-file pd-l2ork -L/home/l2orkist/Downloads/pd-l2ork/emscripten/build/../../libpd/libs -lpd -lm

I also updated deprecated EXTRA_EXPORTED_RUNTIME_METHODS to EXPORTED_RUNTIME_METHODS, as follows:

-s ASYNCIFY -s "ASYNCIFY_IMPORTS=['__Pd_loadLib']" \
-s USE_SDL=2 -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s ALLOW_MEMORY_GROWTH=1 \
-s FORCE_FILESYSTEM=1 -s "EXPORTED_RUNTIME_METHODS=['FS']" \
--no-heap-copy --preload-file pd-l2ork-web -L/home/l2orkist/Downloads/pd-l2ork/emscripten/build/../../libpd/libs -lpd -lm

I also tried changing the way objects are built, for instance:

emcc -s SIDE_MODULE=1 -O3 -sEMULATE_FUNCTION_POINTER_CASTS=1 -s EXPORTED_RUNTIME_METHODS=_average_setup -o "average.wasm" "average.o"

Note the EXPORTED_RUNTIME_METHODS. Not sure if these should be prepended with an underscore. I tried both with or without it. Also tried:

-s EXPORTED_FUNCTIONS=average_setup

That one generates the following warning:

emcc: warning: EXPORTED_FUNCTIONS is not valid with LINKABLE set (normally due to SIDE_MODULE=1/MAIN_MODULE=1) since all functions are exported this mode.  To export only a subset use SIDE_MODULE=2/MAIN_MODULE=2 [-Wunused-command-line-argument]

So, I also tried with SIDE_MODULE=2 and EXPORTED_FUNCTIONS with no success. Curiously, this also suggests that all functions are exported. If so, why are they not being found by loadDynamicLibrary call above?

Where does one find the API information on the loadDynamicLibrary? Is this a built-in C function or an emscripten one?

What can be done to be once again able to reference these bundled objects (preferable), or allowing them to be loaded via the GET option (which I imagine is a fallback routine--please correct me if I may be wrong)?

I would greatly appreciate any input you may have here, as I am completely stumped since the only moving variable is the emscripten version (apart from changing the EXPORTED_RUNTIME_METHODS part, where I tried both versions)? Thank you for your time in considering this request.

sbc100 · 2023-07-31T14:42:27Z

@pd-l2ork, it seems likely that the change that broke your solution was #19310. This change moved the filesystem reading part of library loading into native code which is now part of dlopen. The behaviour of loadDynamicLibrary was also changed, on the assumption that this was an internal function with the public interface being dlopen.
My advice would be to switch to using dlopen directly, unless there is some fundamental reason why that doesn't work for for you?

By the way the lines in your binary such as {"filename":"/pd-l2ork/extra/maxlib/average.wasm","start":15274628,"end":15276992} are the result of the --preload-file flag.

A few other (unrelated) notes on your command line flags.

You might want to consider switching from --preload-file to --embed-file which can be more efficient and avoids a copy.
The --no-heap-copy flag is not longer needed. Its been the default for a long time now.
The EXPORTED_RUNTIME_METHODS and ASYNCIFY_IMPORTS flags can just take simple comma separated lists without any brackets or quoting, which can make you command line simpler. e.g. simply -sEXPORTED_RUNTIME_METHODS=FS

pd-l2ork · 2023-07-31T14:45:53Z

Thank you @sbc100 for your reply. I think the original issue is that many of these objects are larger than 4KB (see the thread title). Has this been resolved? If so, would you be able to provide a skeleton example function that does the same replacing the __PD_loadlib one? Thank you again for your help.

sbc100 · 2023-07-31T14:54:48Z

Chrome has never supported loading module >4kb synchronously. The only way I know of that works around that would be via preloading of the modules, which --preload-file and --embed-file should both take care of.

See preloadedWasm in library_dylink.js for how this work.

pd-l2ork · 2023-07-31T16:47:26Z

Thanks again @sbc100 . So, now I tried changing code to use dlopen (I tried transplanting the original code by @cuinjune ) and now the system is unable to locate the file. Is dlopen trying to search --preload-file database/list (or whatever it is), or is it trying to locate the file on the local drive where the server is running? If it is going through the list, I have objects that share the same name (but are in different subfolders). How could one distinguish between the two, when the code above shows only referencing the actual filename.wasm? In other words, if, for instance the stored object is:

{"filename":"/pd-l2ork/extra/maxlib/average.wasm","start":15274628,"end":15276992}

should the file to be loaded be referenced as:

average.wasm
pd-l2ork/extra/maxlib/average.wasm or
/pd-l2ork/extra/maxlib/average.wasm (note the starting slash)

And what is this location in reference to? To the folder where the server was originally started?

pd-l2ork · 2023-07-31T16:56:41Z

In follow-up to my previous question, how would dlopen example code look inside js? When I try to compile js code with dlopen function, it says it does not recognize the function. Here's the original JS code using loadDynamicLibrary:

EM_JS(int, __Pd_loadLib, (const char *filename, const char *symname), {
    return Asyncify.handleAsync(async () => {
        try {
            await loadDynamicLibrary(UTF8ToString(filename), {loadAsync: true, global: true, nodelete: true, fs: FS});
            var makeout = Module['_' + UTF8ToString(symname)];
            if (typeof makeout === "function") {
                makeout();
                console.log("...success");
                return 1; // success
            }
            else {
                console.log("...no function found");
                return -1; // couldn't find the function
            }
        }
        catch (error) {
            console.log("...failed");
            console.error(error);
            return 0; // couldn't load the external
        }
    });
});

The C version of dlopen is seen in the top post in this thread:

    void *handle;
    typedef int (*func_t)(int);

    handle = dlopen("side.wasm", RTLD_NOW);

    if (!handle) {
        printf("failed to open the library\n");
        return 0;
    }
    func_t func = (func_t)dlsym(handle, "side");

    if (!func) {
        printf("failed to find the method\n");
        dlclose(handle);
        return 0;
    }
    printf("side module size: %d byte\n", func(1));

Note this is a part of the main, so I am not even sure how this would look as a separate C function, since I am quite sure the handle would disappear at the end of that function call.

So, how would this dlopen code look inside a js call, like the one above that is wrapped into EM_JS?

sbc100 · 2023-07-31T17:01:12Z

I believe with pd-l2ork/extra/maxlib/average.wasm or /pd-l2ork/extra/maxlib/average.wasm should work. In the past did average.wasm previously work? I would be surprised if that was the case.

dlopen is a native call so if you don't need to use an EM_JS function at all to call it. You can, for example, just write a function that calls dlopen + dlsym and then returns that function pointer (i.e. the result of dlsym).

kleisauke · 2023-07-31T17:09:27Z

FWIW, Chrome extended this limit to 8MB in version 115, see commit chromium/chromium@d1a1a8f.

pd-l2ork · 2023-07-31T17:46:00Z

Thank you for the clarification. After further hacking I am now finding that the object is successfully loaded. However, for some reason, the core code that this project is trying to port into a browser environment, seems to continue iterating looking for the object, even though it has supposedly found it. I suspect this is because it needs the pointer...

Also, to clarify, average.wasm was never used and it did not work when I manually entered it, so that part should be ok. More soon...

Thanks, again, for all your help.

pd-l2ork · 2023-07-31T19:38:02Z

@sbc100 @kripken OK, so I was able to verify that the new object is indeed loading the setup function. However, once the setup is loaded, the next is instantiation (average_new function located inside a separate .c file dedicated to this object/plugin that is to be dynamically loaded). What I found out is that if this function is declared as static, I end up having the Uncaught RuntimeError: null function or function signature mismatch error. If I recompile the object with that function not being static, everything works. This is a problem, because there are over a thousand of 3rd party objects, some that may share functions that are named the same, so it would be a really bad idea to keep all those functions in a global space. More so, the same source is used to compiile a native version of the software, which makes an additional argument against removing static declaration of those variables. So, how does one ensure that you can dlopen external object/library/plugin that may have its own static variables that should be exposed within the browser?

sbc100 · 2023-07-31T19:50:14Z

As far as I know there is no way to access at static C function from the DLL. static functions are internal to the object file and are not part of the DLL/shared library symbol table. I believe this is true on standard desktop systems such as linux, macOS and windows too. Are you saying you are able to use dlopen/dlsym to load static functions in some other platform?

pd-l2ork · 2023-07-31T19:57:31Z

This code is actively maintained on desktop and runs on Windows, Linux, and OSX. The average_new function being declared static is no problem because (I think--I am not the original author of this architecture) of the following:

average_setup function builds a list of methods (or functions) object has, and stores them in a class declaration (this is c code, so class is in reality a structure), that can be then referenced later when we may need to instantiate the same object (one or more times). In other words, the setup function takes care of the setting up of the class, and only after that is done is the object instantiated.
its instantiation function is invoked through that method, which is a pointer to the static function

As I mentioned, this works fine on all desktop platforms, so I am unsure why it is not working here.

The same code has also worked just fine on the previous version of emscripten that relied upon loadDynamicLibrary.

sbc100 · 2023-07-31T20:10:12Z

So IIUC there is some kind of setup function that returns at struct containing function pointers? That should work fine. Are you able to run the setup function? What does it return? Perhaps if you should share an example of the exact code that is failing that might help (in as simple a form as possible, if you can).

pd-l2ork · 2023-07-31T20:43:50Z

The setup function runs fine. I know this because when run it prints out a message. The average_new is never reached (I have another printout message on top of it). I wish I could simplify this to make it more easy to share. Alas, this may take me a couple of days just to come up with such an abridged version. The overall code base of this thing is humongous--git suggests over 1.5M lines of code, although some of this is definitely due to a collection of images.

sbc100 · 2023-07-31T22:07:59Z

How is average_new supposed to be called? Is it called via dlsym? If not, then how/when is it called? Is it just a the single setup function looked up via dlsym or are there other?

pd-l2ork · 2023-08-01T01:17:53Z

Here's a general breakdown:

The program (pd-l2ork) reads a text file (a.k.a. patch) that contains visual dataflow code split into visual object creation (think of it as boxes with names, each with particular function), object connection (visual cords connecting those boxes), and encapsulation (boxes grouped into other subpatches). Here, we are only concerned with object creation.

Note that everything I am about to share works perfectly in desktop environments and has worked fine in emscripten release sometime in 2021 using loadDynamicLibrary approach that @cuinjune implemented, as shown in the thread above.

Below are key structures and functions:

Core:

class_new: creates a new class (which is a structure) that contains a list of pointers to various functions inside any object, built-in or dynamically loaded):

t_class *class_new(t_symbol *s, t_newmethod newmethod, t_method freemethod,
    size_t size, int flags, t_atomtype type1, ...)
{
    va_list ap;
    t_atomtype vec[MAXPDARG+1], *vp = vec;
    int count = 0;
    t_class *c;
    int typeflag = flags & CLASS_TYPEMASK;
    if (!typeflag) typeflag = CLASS_PATCHABLE;
    *vp = type1;

    va_start(ap, type1);
    while (*vp)
    {
        if (count == MAXPDARG)
        {
            error("class %s: sorry: only %d args typechecked; use A_GIMME",
                s->s_name, MAXPDARG);
            break;
        }
        vp++;
        count++;
        *vp = va_arg(ap, t_atomtype);
    }
    va_end(ap);
    if (pd_objectmaker && newmethod)
    {
        post("class_new newmethod %lx", newmethod);
            /* add a "new" method by the name specified by the object */
   // this is a creation method assigned to this class based on obtained data
  // pd_objectmaker is t_pd struct in charge of creating new objects and to
  // whom creation requests are sent.
        class_addmethod(pd_objectmaker, (t_method)newmethod, s,
            vec[0], vec[1], vec[2], vec[3], vec[4], vec[5]);
        if (class_loadsym)
        {
                /* if we're loading an extern it might have been invoked by a
                longer file name; in this case, make this an admissible name
                too. */
            char *loadstring = class_loadsym->s_name;
            int l1 = strlen(s->s_name), l2 = strlen(loadstring);
            if (l2 > l1 && !strcmp(s->s_name, loadstring + (l2 - l1)))
                class_addmethod(pd_objectmaker, (t_method)newmethod,
                    class_loadsym,
                    vec[0], vec[1], vec[2], vec[3], vec[4], vec[5]);
        }
    }
    c = (t_class *)t_getbytes(sizeof(*c));
    c->c_name = c->c_helpname = s;
    c->c_size = size;
    c->c_methods = t_getbytes(0);
    c->c_nmethod = 0;
    c->c_freemethod = (t_method)freemethod;
    c->c_bangmethod = pd_defaultbang;
    c->c_pointermethod = pd_defaultpointer;
    c->c_floatmethod = pd_defaultfloat;
    c->c_symbolmethod = pd_defaultsymbol;
    c->c_blobmethod = pd_defaultblob; /* MP 20061226 blob type */
    c->c_listmethod = pd_defaultlist;
    c->c_anymethod = pd_defaultanything;
    c->c_wb = (typeflag == CLASS_PATCHABLE ? &text_widgetbehavior : 0);
    c->c_pwb = 0;
    c->c_firstin = ((flags & CLASS_NOINLET) == 0);
    c->c_patchable = (typeflag == CLASS_PATCHABLE);
    c->c_gobj = (typeflag >= CLASS_GOBJ);
    c->c_drawcommand = 0;
    c->c_floatsignalin = 0;
    c->c_externdir = class_extern_dir;
    c->c_savefn = (typeflag == CLASS_PATCHABLE ? text_save : class_nosavefn);

    classtable_register(c);
    post("class:%s newmethod=%lx", c->c_name->s_name, newmethod);
    return (c);
}

Core or external objects:

object_setup function (each object has its own name, e.g. average_setup, or something_setup) that is called once the object is found (either built-in or dlopen-ed, the desktop version also uses dlopen). Below is an example of the average_setup:

void average_setup(void)
{
   // this creates the class, passing it creation function, freeing function, and optional arguments
    average_class = class_new(gensym("average"), (t_newmethod)average_new,
    	(t_method)average_free, sizeof(t_average), 0, A_DEFFLOAT, 0);
   // I added this to post the function address
    post("average_setup %lx", (t_method)average_new);
  // this adds additional methods, all of which are statically defined
    class_addmethod(average_class, (t_method)average_reset, gensym("reset"), 0);
    class_addmethod(average_class, (t_method)average_linear, gensym("linear"), 0);
    class_addmethod(average_class, (t_method)average_geometric, gensym("geometric"), 0);
    class_addmethod(average_class, (t_method)average_weight, gensym("weight"), 0);
    class_addfloat(average_class, average_float);
	class_addmethod(average_class, (t_method)average_index, gensym("index"), A_FLOAT, 0);

   // version is a const char text that is visible when I run my code, so the code gets this far
    logpost(NULL, 4, version);
}

Here's the average_new function that, like all other objects, returns pointer to its instance

static void *average_new(t_floatarg f)
{
   // this first post that should print to console is never reached
	post("average_new");
	t_average *x = (t_average *)pd_new(average_class);
	x->x_inindex = inlet_new(&x->x_ob, &x->x_ob.ob_pd, gensym("float"), gensym("index"));
	x->x_outfloat = outlet_new(&x->x_ob, gensym("float"));
	x->x_outtendency = outlet_new(&x->x_ob, gensym("float"));

		/* zero out the array */
	x->x_index = (t_int)f;
	if (x->x_index < 1)
		x->x_index = 1;
	x->x_input = (t_float *)calloc(x->x_index,sizeof(t_float));

	x->x_inpointer = 0;
	x->x_average = 0;
	x->x_mode = 0;
    return (void *)x;
}

function responsible for loading the function and then running the setup:

static int sys_do_load_lib(t_canvas *canvas, const char *objectname,
    const char *path)
{
 // cutting bunch of irrelevant stuff
 // ...
// this is what it used to be and what used to call EM_JS __Pd_loadLib call that relied on the loadDynamicLibrary
// now it is commented out
/*
#ifdef __EMSCRIPTEN__
    post("__Pd_loadLib <%s> <%s>", filename, symname);
    int res = __Pd_loadLib(filename, symname);
    if (ret == 1) {
        post("...success");
        (*makeout)();
        class_set_extern_dir(&s_);
        return (1);
    }
    else if (ret == 0) {
        verbose(1, "%s: couldn't load", filename);
        post("...couldn't load");
        class_set_extern_dir(&s_);
        return (0);
    }
    else { //ret is -1, object was opened but no setup function was found
        post("...no setup function found");
        return(0);
    }
#elif _WIN32
*/
#ifdef _WIN32
   // this part is cut, as it does not apply
#elif defined(HAVE_LIBDL) || defined(__FreeBSD__) || defined(__EMSCRIPTEN__) // this is where I've now added EMSCRIPTEN part
   // this is what we use currently
    post("sys_do_load_lib <%s> <%s>", filename, symname);
    dlobj = dlopen(filename, RTLD_NOW | RTLD_GLOBAL);
    if (!dlobj)
    {
        error("%s: %s", filename, dlerror());
        class_set_extern_dir(&s_);
        return (0);
    }
    makeout = (t_xxx)dlsym(dlobj,  symname);
    if(!makeout)
        makeout = (t_xxx)dlsym(dlobj,  "setup");
#else
#warning "No dynamic loading mechanism specified, \
    libdl or WIN32 required for loading externals!"
#endif

    if (!makeout)
    {
        error("load_object: Symbol \"%s\" not found", symname);
        class_set_extern_dir(&s_);
        return 0;
    }
    (*makeout)();
    class_set_extern_dir(&s_);
    post("...sys_do_load_lib success");
    return (1);
}

When creating objects that come built-in with the core pd-l2ork package (unlike 3rd party objects), everything is created just fine. However, when trying to create dynamically loaded object, this is something that used to work fine around 2021 and now does not, with only notable change between now and then being emscripten version update. So, I set out to adapt loading using dlopen and here's the current status:

Core pd-l2ork reads the object creation request from the patch file. It searches the existing class list and if it does not find it, tries to look for an external. This eventually triggers sys_do_load_lib, which in turn triggers average_setup, which triggers class_new, and then invokes creation of the actual object.

The resulting relevant console printout excerpt shows what worked and where it fails:

// this is the call when pd-l2ork does not know what to instantiate, so it calls this to trigger search
// (1 stands for a number of arguments, in this case it is 10. this means that the object will be
// giving a running average over the last 10 numbers
main.js:298 new_anything <maxlib/average> 1
// here you see the call with proper location and setup function
main.js:298 sys_do_load_lib <pd-l2ork-web/extra/maxlib/average.wasm> <average_setup>
// here we see the address of the new method which is 2674
main.js:298 class_new newmethod 2674
// here we see methods being added (we have both the long and short version here,
// so both versions work)
main.js:298 addmethod total=297 <average>
main.js:298 addmethod total=298 <maxlib/average>
// here we can see that class_new newmethod has indeed correct address,
// matching the one above
main.js:298 class:average newmethod=2674
// here we see what the address of the newmethod is coming from the average_setup
// this also proves that average_setup is found and is triggered
main.js:298 average_setup 2674
// here you see how other methods (static functions inside average.c file) are being added,
// so that they can be referenced, so this tells us class has indeed gotten new methods
main.js:298 addmethod total=1 <reset>
main.js:298 addmethod total=2 <linear>
main.js:298 addmethod total=3 <geometric>
main.js:298 addmethod total=4 <weight>
main.js:298 addmethod total=5 <index>
// this one is also called from average_setup via logpost (level 4 in terms of verbosity)
main.js:298 verbose(4): average v0.2, written by Olaf Matthes <[email protected]>,
revised by Ivica Ico Bukvic <[email protected]>
// this tells us that sys_do_load_lib has successfully completed
main.js:298 ...sys_do_load_lib success
// now that the class has been properly created, we call new_anything again
// this time it should be able to create an object because it has its class in its class list
main.js:298 ...new_anything typedmess <maxlib/average> 1
// unfortunately, this is what happens if average_new is a static function (typedmess is
// simply resending the patch text asking to instantiate the object)
// if I make average not static, the object instantiates fine but that is not an option,
// as doing so will very likely create address clashes among over a thousand of 3rd party objects
main.wasm:0x1359d1 Uncaught (in promise) RuntimeError: null function or function
signature mismatch
    at main.wasm:0x1359d1
    at main.wasm:0x135344
    at main.wasm:0x13581d
    at main.wasm:0x130275
    at main.wasm:0x12169d
    at main.wasm:0x12155d
    at main.wasm:0x13557b
    at main.wasm:0x130275
    at main.wasm:0x133a73
    at main.wasm:0x133b0e
$pd_typedmess @ main.wasm:0x1359d1
$new_anything @ main.wasm:0x135344
$pd_typedmess @ main.wasm:0x13581d
$binbuf_eval @ main.wasm:0x130275
$func1654 @ main.wasm:0x12169d
$canvas_obj @ main.wasm:0x12155d
$pd_typedmess @ main.wasm:0x13557b
$binbuf_eval @ main.wasm:0x130275
$binbuf_evalfile @ main.wasm:0x133a73
$glob_evalfile @ main.wasm:0x133b0e
$libpd_openfile @ main.wasm:0x187a73
$_ZN2Pd9openPatchERKNSt3__212basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEES8_ @ main.wasm:0xaa48b
$_ZN10emscripten8internal13MethodInvokerIM2PdFvRKNSt3__212basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEESB_EvPS2_JSB_SB_EE6invokeERKSD_SE_PNS0_11BindingTypeIS9_vEUt_ESL_ @ main.wasm:0xa920a
Pd$openPatch @ VM121:10
openPatch @ main.js:2094
init @ main.js:2153
await in init (async)
mainInit @ main.js:840
__mainInit @ main.js:1
$__original_main @ main.wasm:0xa8817
$main @ main.wasm:0xa9488
callMain @ main.js:1
doRun @ main.js:1
run @ main.js:1
runCaller @ main.js:1
removeRunDependency @ main.js:1
processPackageData @ main.js:1
(anonymous) @ main.js:1
xhr.onload @ main.js:1
load (async)
fetchRemotePackage @ main.js:1
loadPackage @ main.js:1
(anonymous) @ main.js:1
(anonymous) @ main.js:1

Below is the relevant build output:

average.c:

emcc -DPD -I/home/l2orkist/Downloads/pd-l2ork/emscripten/../pd/src -Wall -W -Wno-unused-parameter -s SIDE_MODULE=1 -O3 -o "average.o" -c "average.c"
average.c:218:22: warning: format string is not a string literal (potentially insecure) [-Wformat-security]
  218 |     logpost(NULL, 4, version);
      |                      ^~~~~~~
average.c:218:22: note: treat the string as an argument to avoid this
  218 |     logpost(NULL, 4, version);
      |                      ^
      |                      "%s", 
1 warning generated.
emcc -s SIDE_MODULE=1 -O3 -o "average.wasm" "average.o"   
chmod a-x "average.wasm"

main (combination of core and all bundled externals):

emcc -I/home/l2orkist/Downloads/pd-l2ork/emscripten/build/../../pd/src -I/home/l2orkist/Downloads/pd-l2ork/emscripten/build/../../libpd/libpd_wrapper -O3 -s MAIN_MODULE=1 --bind -o main.html /home/l2orkist/Downloads/pd-l2ork/emscripten/build/../../emscripten/src/main.cpp /home/l2orkist/Downloads/pd-l2ork/emscripten/build/../../emscripten/src/pd.cpp /home/l2orkist/Downloads/pd-l2ork/emscripten/build/../../emscripten/src/js.cpp \
-s USE_SDL=2 -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s ALLOW_MEMORY_GROWTH=1 \
-s FORCE_FILESYSTEM=1 -s EXPORTED_RUNTIME_METHODS=FS \
--preload-file pd-l2ork-web -L/home/l2orkist/Downloads/pd-l2ork/emscripten/build/../../libpd/libs -lpd -lm

sbc100 · 2023-08-01T01:46:21Z

Regarding the old EM_JS code, I think I do see a problem there. The code is expecting symbols from the loaded library to be presend on the global Module object with the underscore prefix. i.e. var makeout = Module['_' + UTF8ToString(symname)];

These days we use wasmImports as the global symbol table. So that line should be replaced with var makeout = wasmImports[UTF8ToString(symname)];. That is, assuming that loadDynamicLibrary succeeds.

However, the dlopen approach should also work and be simpler and more robust over time. Are the dlsym calls succeeding? Presumable one of the gensym calls is failing? What does gensym do exactly? Does it call dlsym?

sbc100 · 2023-08-01T01:47:21Z

If you want to get better stack traces can you try building with -g (or just --profiling-funcs). That should give function names rather than just:

signature mismatch
    at main.wasm:0x1359d1
    at main.wasm:0x135344
    at main.wasm:0x13581d
    at main.wasm:0x130275
    at main.wasm:0x12169d
    at main.wasm:0x12155d
    at main.wasm:0x13557b
    at main.wasm:0x130275
    at main.wasm:0x133a73
    at main.wasm:0x133b0e

pd-l2ork · 2023-08-01T04:17:24Z

These days we use wasmImports as the global symbol table. So that line should be replaced with var makeout = wasmImports[UTF8ToString(symname)];. That is, assuming that loadDynamicLibrary succeeds.

Thank you. I will try to revert back to that implementation and see how it goes.

However, the dlopen approach should also work and be simpler and more robust over time. Are the dlsym calls succeeding? Presumable one of the gensym calls is failing? What does gensym do exactly? Does it call dlsym?

gensym is simply generating a symbol (string) that is kept in a string table. This is needed for a number of reasons, including wirelessly connecting objects via internal send and receive objects (among other things).

BTW, dlopen does work up through the setup function. Where it fails is when trying to instantiate average_new, which should've been stored in class_new, and whose address, per my debug printouts, appears to be tracking across multiple function calls.

pd-l2ork · 2023-08-01T04:17:58Z

If you want to get better stack traces can you try building with -g (or just --profiling-funcs). That should give function names rather than just:
signature mismatch
    at main.wasm:0x1359d1
    at main.wasm:0x135344
    at main.wasm:0x13581d
    at main.wasm:0x130275
    at main.wasm:0x12169d
    at main.wasm:0x12155d
    at main.wasm:0x13557b
    at main.wasm:0x130275
    at main.wasm:0x133a73
    at main.wasm:0x133b0e

I will also make sure to compile with a debug flag and will let you know. Thank you.

pd-l2ork · 2023-08-01T05:21:04Z

So, I am completely baffled. I just went to enable the debug options and I don't think I even did it on all components that comprise the final main wasm file and upon recompiling dlopen worked as expected. I then, went on to disable debugging to revert everything back and was unable to break it again. Eventually, I was able to enable debugging for everything and that ended-up spawning some odd errors, but this is a separate issue at this point. I know for a fact that I cleaned the entire tree before. Could it be that there may have been a stale .o file that could've caused this? I am thinking .o file from a different compile time may generate a different signature, resulting in an incompatible build perhaps? I am at a loss, and my brain is fried... More tomorrow... Thanks again for all your help.

pd-l2ork · 2023-08-01T19:39:00Z

For whatever rason I am now unable to break it in the way it was broken before, but am now running into the debug-enabled version throwing bunch of errors that are not present in an optimized build. Below is an example of the output. Everything loads fine in the browser, but as son as I click, I get the following stream of errors (they repeat, so below is only a snapshot). None of this is a problem when running -O3 build (this one is -g):

main.js:56 Aborted(Assertion failed: exceptfds not supported)
printErr @ main.js:56
abort @ main.js:950
assert @ main.js:631
___syscall__newselect @ main.js:5420
$select @ main.wasm:0x28f3a7
$sys_domicrosleep @ main.wasm:0x15ce2e
$sys_pollgui @ main.wasm:0x15eb39
$libpd_process_float @ main.wasm:0x1aa664
$Pd::audioOut(unsigned char*, int) @ main.wasm:0xb90db
$Pd::forwardAudioOut(void*, unsigned char*, int) @ main.wasm:0xb6c3b
$HandleAudioProcess @ main.wasm:0x209500
dynCall @ main.js:1622
SDL2.audio.scriptProcessorNode.onaudioprocess @ main.js:1287
main.js:969 Uncaught RuntimeError: Aborted(Assertion failed: exceptfds not supported)
    at abort (main.js:969:11)
    at assert (main.js:631:5)
    at ___syscall__newselect (main.js:5420:7)
    at select (main.wasm:0x28f3a7)
    at sys_domicrosleep (main.wasm:0x15ce2e)
    at sys_pollgui (main.wasm:0x15eb39)
    at libpd_process_float (main.wasm:0x1aa664)
    at Pd::audioOut(unsigned char*, int) (main.wasm:0xb90db)
    at Pd::forwardAudioOut(void*, unsigned char*, int) (main.wasm:0xb6c3b)
    at HandleAudioProcess (main.wasm:0x209500)
abort @ main.js:969
assert @ main.js:631
___syscall__newselect @ main.js:5420
$select @ main.wasm:0x28f3a7
$sys_domicrosleep @ main.wasm:0x15ce2e
$sys_pollgui @ main.wasm:0x15eb39
$libpd_process_float @ main.wasm:0x1aa664
$Pd::audioOut(unsigned char*, int) @ main.wasm:0xb90db
$Pd::forwardAudioOut(void*, unsigned char*, int) @ main.wasm:0xb6c3b
$HandleAudioProcess @ main.wasm:0x209500
dynCall @ main.js:1622
SDL2.audio.scriptProcessorNode.onaudioprocess @ main.js:1287
main.js:56 Aborted(Assertion failed: exceptfds not supported)
printErr @ main.js:56
abort @ main.js:950
assert @ main.js:631
___syscall__newselect @ main.js:5420
$select @ main.wasm:0x28f3a7
$sys_domicrosleep @ main.wasm:0x15ce2e
$sys_pollgui @ main.wasm:0x15eb39
$libpd_process_float @ main.wasm:0x1aa664
$Pd::audioOut(unsigned char*, int) @ main.wasm:0xb90db
$Pd::forwardAudioOut(void*, unsigned char*, int) @ main.wasm:0xb6c3b
$HandleAudioProcess @ main.wasm:0x209500
dynCall @ main.js:1622
SDL2.audio.scriptProcessorNode.onaudioprocess @ main.js:1287
main.js:969 Uncaught RuntimeError: Aborted(Assertion failed: exceptfds not supported)
    at abort (main.js:969:11)
    at assert (main.js:631:5)
    at ___syscall__newselect (main.js:5420:7)
    at select (main.wasm:0x28f3a7)
    at sys_domicrosleep (main.wasm:0x15ce2e)
    at sys_pollgui (main.wasm:0x15eb39)
    at libpd_process_float (main.wasm:0x1aa664)
    at Pd::audioOut(unsigned char*, int) (main.wasm:0xb90db)
    at Pd::forwardAudioOut(void*, unsigned char*, int) (main.wasm:0xb6c3b)
    at HandleAudioProcess (main.wasm:0x209500)
abort @ main.js:969
assert @ main.js:631

I tried compiling with -s USE_PTHREADS as one of the online resources suggested, but that made things even worse, making the program fail to even load (threw one error that I don't have on hand and stopped right there).

Should I make this a separate issue?

pd-l2ork · 2023-08-01T20:36:33Z

Here is a more on-topic question. So, when I run a patch (a snippet of code) that does not require dlopen, everything works fine. However, if I have a patch that invokes dlopen AND the object in question is supposed to process audio, even though the object now loads correctly, as soon as I do anything on the page with my mouse (e.g. click onto a visual widget, a slider or a button), I get the following error:

main.wasm:0x1342f3 Uncaught RuntimeError: memory access out of bounds
    at main.wasm:0x1342f3
    at main.wasm:0x188d52
    at main.wasm:0xaa597
    at main.wasm:0xa9335
    at ClassHandle.Pd$sendFloat [as sendFloat] (eval at newFunc (main.js:1:396680), <anonymous>:10:1)
    at gui_slider_bang (main.js:1407:15)
    at gui_slider_onmousemove (main.js:1440:9)
    at main.js:1615:9

The source of error may be different depending on what I pressed, but the end result is always the same. I tried recompiling with -s TOTAL_MEMORY=128MB instead of -s ALLOW_MEMORY_GROWTH=1 thinking perhaps resizing the heap may screw up addresses which are stored in the class, in case the entire thing is repositioned to a different memory address, but that made no difference.

sbc100 · 2023-08-01T20:44:51Z

Looks like a memory corruption issue. I would recommend a debug build with -sASSERTIONS and --profiling-funcs. That might. You might also want to try -fsanitize=address. It sounds like we are getting off topic for this issue though.

sbc100 · 2023-08-01T20:46:06Z

Regarding the original issue: Chome just increased the limit from 4kb to 8Mb
chromium/chromium@d1a1a8f. So this issue should go away once that hits chrome stable.

pd-l2ork · 2023-08-02T04:54:46Z

So, as a closure to this discussion, I learned two critical things:

fixed exceptfds issue by updating the source (see Bunch of Aborted(Assertion failed: exceptfds not supported) errors when running unoptimized code #19951 for more info)
learned that -fsanitize=address is the magical thing that solves all my problems. If I don't use this compile flag, I get random out of memory errors that happen all over the code, which is unusual for a fairly mature source base that has been running on other platforms for years. Once I enabled this flag, both -g and -O3 builds run just fine. I imagine perhaps this flag makes address handling stricter, which is essential for the way m_class functions, as described above. Is this correct? Or, could this be an emscripten bug of sorts? In other words, why not make this default?

Thanks @sbc100 again for all your time and assistance.

liquidaty mentioned this issue Feb 6, 2023

Clarification of license for site using web assembly + static linking? libgit2/libgit2#6467

Closed

pd-l2ork mentioned this issue Aug 2, 2023

Bunch of Aborted(Assertion failed: exceptfds not supported) errors when running unoptimized code #19951

Closed

Opening a side module larger than 4KB fails on Chrome #11753

Opening a side module larger than 4KB fails on Chrome #11753

Comments

cuinjune commented Jul 29, 2020

gerald-dotcom commented Jul 29, 2020

kripken commented Jul 29, 2020

cuinjune commented Jul 30, 2020 • edited Loading

sbc100 commented Jul 30, 2020

cuinjune commented Jul 31, 2020

sbc100 commented Jul 31, 2020

cuinjune commented Jul 31, 2020 • edited Loading

cuinjune commented Jul 31, 2020

sbc100 commented Jul 31, 2020

cuinjune commented Jul 31, 2020

cuinjune commented Jul 31, 2020

gerald-dotcom commented Jul 31, 2020

cuinjune commented Jul 31, 2020 • edited Loading

sbc100 commented Aug 1, 2020

cuinjune commented Aug 1, 2020

sbc100 commented Aug 1, 2020

cuinjune commented Aug 1, 2020

sbc100 commented Aug 1, 2020

sbc100 commented Aug 1, 2020

cuinjune commented Aug 1, 2020

sbc100 commented Aug 1, 2020

cuinjune commented Aug 1, 2020

sbc100 commented Aug 1, 2020

cuinjune commented Aug 1, 2020 • edited Loading

sbc100 commented Aug 1, 2020

sbc100 commented Aug 1, 2020

cuinjune commented Aug 1, 2020

sbc100 commented Aug 1, 2020

cuinjune commented Aug 1, 2020 • edited Loading

sbc100 commented Aug 1, 2020

AndyC-NN commented Feb 20, 2022 • edited Loading

aisnote commented Sep 2, 2022

Seems it is caused by my.wasm building settings?

pd-l2ork commented Jul 31, 2023 • edited Loading

sbc100 commented Jul 31, 2023

pd-l2ork commented Jul 31, 2023

sbc100 commented Jul 31, 2023

pd-l2ork commented Jul 31, 2023

pd-l2ork commented Jul 31, 2023

sbc100 commented Jul 31, 2023

kleisauke commented Jul 31, 2023

pd-l2ork commented Jul 31, 2023 • edited Loading

pd-l2ork commented Jul 31, 2023 • edited Loading

sbc100 commented Jul 31, 2023

pd-l2ork commented Jul 31, 2023 • edited Loading

sbc100 commented Jul 31, 2023

pd-l2ork commented Jul 31, 2023

sbc100 commented Jul 31, 2023

pd-l2ork commented Aug 1, 2023 • edited Loading

sbc100 commented Aug 1, 2023

sbc100 commented Aug 1, 2023

pd-l2ork commented Aug 1, 2023

pd-l2ork commented Aug 1, 2023

pd-l2ork commented Aug 1, 2023

pd-l2ork commented Aug 1, 2023

pd-l2ork commented Aug 1, 2023

sbc100 commented Aug 1, 2023

sbc100 commented Aug 1, 2023

pd-l2ork commented Aug 2, 2023 • edited Loading

cuinjune commented Jul 30, 2020 •

edited

Loading

cuinjune commented Jul 31, 2020 •

edited

Loading

cuinjune commented Jul 31, 2020 •

edited

Loading

cuinjune commented Aug 1, 2020 •

edited

Loading

cuinjune commented Aug 1, 2020 •

edited

Loading

AndyC-NN commented Feb 20, 2022 •

edited

Loading

pd-l2ork commented Jul 31, 2023 •

edited

Loading

pd-l2ork commented Jul 31, 2023 •

edited

Loading

pd-l2ork commented Jul 31, 2023 •

edited

Loading

pd-l2ork commented Jul 31, 2023 •

edited

Loading

pd-l2ork commented Aug 1, 2023 •

edited

Loading

pd-l2ork commented Aug 2, 2023 •

edited

Loading