Event Loop Integration: Support for Poll, Epoll or Coroutines

[femski]

I am trying to port Mapbox GL Native (https://github.com/mapbox/mapbox-gl-native) to WASM and final piece is getting event loop to work in emscripten/wasm.

Each thread in mapbox runs its own event loop - blocking on an empty file descriptor. As new events arrives loop is woken up by signalling on the file descriptor.

I asked libuv folks libuv/help#128 if I could get this to work under emscripten and answer was likely no since emscripten does not have real "poll" (see #2590) or epoll (#5033):

So whats the alternative?

I have tried using emscripten_pause_main_loop - but don't understand how to wakeup/resume main loop via async task once paused. All the sleep and yield functions are gone. ASINCIFY has been removed and fiber.h introduced but there are no high level constructs like coroutines anymore to create embedded event loop yet.

Will epoll or poll supported anytime soon? Will you reintroduce coroutines or provide samples of how to fiber.h to create an event loop like mechanism?

thank you.

[kripken]

ASINCIFY has been removed

The old Asyncify was removed, but the new backend has a new and better Asyncify, https://emscripten.org/docs/porting/asyncify.html We can build things like coroutines from that, and should be able to support things like poll etc. (but that would also require filesystem work, aside from the sync/async issue that Asyncify fixes).

I'm not aware of anyone concretely working on that, but contributions would be very welcome! If someone tries and has Asyncify questions I can help with those.

[Akaricchi]

Will you reintroduce coroutines or provide samples of how to fiber.h to create an event loop like mechanism?

I'm not sure if this is actually what you're looking for, but I have a little project here that implements stackful coroutines, which work much like emscripten's old coroutines. The Fibers API is used for the Emscripten backend.

I'm also the author of the fibers api, so I can help with that if you have any questions, but I suspect it's not actually what you're looking for.

[femski]

@Akaricchi @kripken

Thank you for your help ! I was able to make some progress. Basic maps are working using nothing but condition variables and mutex locks. See:

https://avnav.com/mbgl

Now, trying to figure how to wakeup one asyncify paused thread from another wasm thread:
#10515

thanks again.

[femski]

It works! I had to hack a replacement for libuv - its not pretty but you can see Mapbox GL native running in browser today at above link.

We are wasting CPU, threads and memory - but for today I am happy that such a complex system can be ported to WASM thanks to Emscripten.

I would be great to have polling on empty file handles (or cross thread jumps via Fribers/asyncify). This will improve performance to be better than the Javscript version of the same library. What can we do make this happen ?

Some more details of this port are here:
mapbox/mapbox-gl-js#4835 (comment)

[kripken]

@femski Does the hack you implemented use asyncify (with or without fibers)? I'm not sure if you're asking whether something can be upstreamed (probably yes!) or whether there are open technical questions.

[femski]

@kripken

Hack I implemented was to keep a thread waiting on spinlock - it executes commands sent by main thread. This does not use asyncify (because you can not suspend one thread and un-suspend it from another wasm thread using asyncify). I did not use fibers either (because fibers and coroutines also currently can't be suspended on one thread and be resumed form another).

I would now like to use fibers to accomplish this. It will be much faster and far more elegant.

@Akaricchi suggests how we could make asyncify and fibers jump threads:
#10515 (comment)

I don't think I have enough understanding of Emscripten internals or fibers to make this happen. But then I also thought I could not port Mapbox to WASM. So I am available to do anything needed here to make this happen. This will make Mapbox and many other programs feasible (and faster) on WASM.

[femski]

Problem we are trying to solve is like this:

https://github.com/ivovandongen/cpp-runloop/blob/master/src/basic_runloop_spin_lock/runloop.cpp

We have thread that does a few things and then goes to sleep - waiting on a pipe. Its woken up by another thread to resume and process more tasks queued while it was sleeping.

We want to do this using asyncify/fibers - so there is probably no waiting on empty pipe/file handle but something else (a piece of memory ?).

[kripken]

Yeah, maybe one way to do that might be to have a "runtime" written in C that handles this. The runtime would be on a single thread, and other threads can wait for it using the normal pthread mechanisms. And that runtime itself might use asyncify.

[stale]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant.