Switch to a proper scheduler

[rgrinberg]

The current situation suffers from poor performance and consumes too much resources. I'm planning to get rid of it with libev bindings I've developed: https://github.com/rgrinberg/lev.

This work has to be done in steps:

• Update to the latest Fiber api in dune.
• Implement a general fiber based scheduler for Lev.
• Replace the current home made scheduler with it.
• Test on windows

Once this is done, we can even consider removing the thread used for merlin. I'm not 100% convinced it buys us something at this point.

[ulugbekna]

Couldn't we switch to lwt? It would lower the entry barrier by far imho. We've had several scheduler-related problems, and the current scheduler is a dreadful piece of code with signals, threads, etc. again imho.

[rgrinberg]

How would it lower the barrier? Fiber is far simpler than Lwt (and faster, and with much better error handling).

We've had plenty of scheduler related problems because we've build the scheduler wrong. Fortunately, this final approach should fix the issue for good by building on top of reliable primitives. Also, dune itself suffers from the same problems so if the libev based scheduler performs well, we'll be able to use it in dune too. For dune, Lwt is definitely out of question so I wouldn't save myself any time by avoiding this work here in lsp.

[rgrinberg]

Once this is done, we can even consider removing the thread used for merlin. I'm not 100% convinced it buys us something at this point.

This is unfortunately impossible as merlin spawns ppx processes which can be quite slow. It will have to continue running in a separate thread.

But otherwise, I've completed the ported and it seems to be working quite well.