Using findlib.dynload shouldn't imply -linkall for executables.

[ejgallego]

This is a follow up from #4348

Take the project at https://github.com/ejgallego/dune-bug-dynload

I see no reason why adding findlib.dynload to the (libraries ...) field should imply -linkall. Moreover, this can be problematic for some libraries, and tends to create huge executables.

cc: @bobot

[bobot]

Moreover, this can be problematic for some libraries, and tends to create huge executables.

It is true, but it is more general than -linkall, ocaml binaries are big. #3866 describes possibilities to fix that problem globally.

I see no reason why adding findlib.dynload to the (libraries ...) field should imply -linkall.

My reason is that I don't know how to give the possibility to load any library reliably without using -linkall as I described in #4348. I'm open to suggestion for doing differently.

(In the readme of the project as example the last line Hello, World, from fl1! -> Hello, World, from fl2!)

[bobot]

NB: ocamlfind adds -linkall when linking findlib.dynload.

[ejgallego]

My reason is that I don't know how to give the possibility to load any library reliably without using -linkall as I described in #4348.

Maybe we can discuss this in the call, or sorry if I'm slow, but where is linkall needed exactly?

Is it because you need to initialize the table with modules present in the executable?

The way I understand it, an executable with plugins will call Fl_dynload.load_packages , which will check what has been loaded, and then Dynlink the missing object files. I don't see how -linkall interacts with that, unless indeed you need to ensure fully compilation units present in the executable. If that's the case, then this would be an upstream (OCaml) bug.

cc: @gerdstolpmann as per the -linkall comment for ocamlfind.

[bobot]

which will check what has been loaded

You should define the "what":

• libraries? easy to compute since ocamlfind or dune knows the one it wanted to link, but requires -linkall to be full
• modules? harder to compute since it depends on the code of the executable, parsing of the ocamlobjinfo output of the cmx used could help.

then Dynlink the missing object files

Currently we have .cmxs of whole library not of the independent modules, and dynlink fail if a module is already linked. So we can't dynlink missing modules of a partially statically linked library.

[ejgallego]

Currently we have .cmxs of whole library not of the independent modules, and dynlink fail if a module is already linked. So we can't dynlink missing modules of a partially statically linked library.

I see thanks, so that's the (only) reason then we need to use -linkall; is my understanding correct?

[bobot]

Yes, but still finding which modules are really linked is not trivial either, just less impossible currently. On the non technical part, I mentioned previously that linking part of a library later can create subtile bugs because the developers of a library would have to take into account different execution order of its module units.

[ejgallego]

I guess if the library does support that, they authors will have to provide some kind of commutativity.

That's actually the reason I was trying to get rid of linkall in all my developments [in particular for Coq] because it is pretty fragile as of today.

I will upstream the discussion to see what happens, but my feeling is that likely this will end on a documentation issue.

[bobot]

That's actually the reason I was trying to get rid of linkall in all my developments [in particular for Coq] because it is pretty fragile as of today.

Can you elaborate why it is fragile? For me, using linkall for plugins makes things more robust because we are using them in only one way, not many.

[SkySkimmer]

Is there a way to disable the auto linkall? For instance (link_flags \ -linkall) is accepted but does nothing

[ejgallego]

We could add that, but can you explain a bit more what you are trying to do @SkySkimmer ?

Note that if you don't use -linkall, you can as well just drop findlib.dynload as it will be unsound (there is extra discussion in some Coq PRs about removing the use of -linkall)

The problem comes because findlib / the build system won't understand what modules were effectively linked into an executable. Thus imagine, you have library parrot with modules A B, but foo.exe on refers to A, so foo.exe won't include B.

Then you have plugin Bar which depends on parrot refers to both A B. Unfortunately, when foo.exe tries to dynload Bar, Dynlink will fail as B is missing from foo.

[SkySkimmer]

Dropping dynload only works if you have sufficiently fine grained libraries.
ie instead of doing coq/coq#16678 it should have been possible to drop -linkall from coqworkmgr knowing that it will never actually call dynload (and dynload would not even get linked)

[ejgallego]

Yes, the more knowledge you have the more precise you can be with dependencies. In this all build systems are always over-approximations.

I think the solution we did for coq/coq#16678 is much better than hacking the flags in this way which requires manual code inspection and maintenance.