conda-installed gfortran on OSX: builds fail at runtime

[brian-rose]

I distribute a Python package that includes f2py-compiled Fortran extensions. I want to use conda to package this (for my students and other users) as a self-contained environment including gfortran. But I'm having trouble getting anything to compile and run on Mac OSX 10.11 with the conda-installed version of gfortran.

A simple HelloWorld example fails at runtime:

conda create --name=test gcc
source activate test

This then becomes the default gfortran version:

(test)suggestionbox_new:temp br546577$ which gfortran
/Users/br546577/anaconda/envs/test/bin/gfortran

I then compile and try to run a 'hello world' program:

(test)suggestionbox_new:temp br546577$ gfortran hello.f90
(test)suggestionbox_new:temp br546577$ ./a.out
dyld: Library not loaded: @rpath/./libgfortran.3.dylib
  Referenced from: /Users/br546577/Desktop/temp/./a.out
  Reason: image not found
Trace/BPT trap: 5

The same test works without error on linux.

I've found several other reports of such library errors with conda and gfortran on OSX, e.g. #498, ReactionMechanismGenerator/PyDAS#8

It looks to me like the version of gfortran installed by conda is completely unusable.

[astrofrog]

I'm seeing this issue too!

[mingwandroid]

Well, it's the same rpath GCC flag that is common for GCC and gfortran on both Linux and OS X. This should work for you:

gfortran ~/gd.cio/hello.f90 -Wl,-rpath,${CONDA_PREFIX}/lib

.. alternatively you can post process the executable as follows:

install_name_tool -change @rpath/./libgfortran.3.dylib ${CONDA_PREFIX}/lib/libgfortran.3.dylib -change @rpath/./libquadmath.0.dylib ${CONDA_PREFIX}/lib/libquadmath.0.dylib a.out

You used to be able to use DYLD_* variables (specifically DYLD_FALLBACK_LIBRARY_PATH) to modify the behaviour of dyld, but Apple decided to reduce their attack vector surface area and removed them with the introduction of SIP.

[mingwandroid]

I'm closing this as it's just how these things work. If you have any suggestions for improving it feel free to reopen.

The reason it works on Linux is (probably!) because on that OS, ld.so will fallback to finding /usr/lib/libgfortran.so which will be more recent and backwards compatible.

[astrofrog]

@mingwandroid - just out of curiosity, why does this not happen with the gcc compiler?

[mingwandroid]

Because on OS X, the standard C library is implemented in libSystem.dylib and that is present on all systems in /usr/lib/libSystem.dylib.

A default fortran standard library doesn't ship with the OS so we must provide one.

Conda's dynamic libraries (on OS X and Linux) are always identified via relative paths so that our environments are trivially relocatable (that is a requirement for hardlinks to work, and is just nicer than the alternative). You can see the functions used to enforce this here:

https://github.com/conda/conda-build/blob/master/conda_build/post.py#L264-L307

[brian-rose]

Thanks @mingwandroid for the info! I was able to implement a workaround for OSX in my package that depends on F2PY builds, using your install_name_tool solution on the compiled *.so objects:
https://github.com/brian-rose/climlab/pull/41

[rainwoodman]

@mingwandroid

Sorry for bring this up again.. but should the cross compilation tools add the rpath flag to LDFLAGS by default on osx in the activate script of gfortran? Currently the configure script of openmpi fails because gfortran won't produce a runnable executable for this error.

[mingwandroid]

Personally I do not think they should. If a recipe needs this stuff (RPATH, -L, -I) then it should add this stuff.

However I've been voted down on this so will go along with the majority opinion and will accept such a change in a PR.

[msarahan]

The discussion of this issue is at conda-forge/conda-forge.github.io#460 (comment)

You'd need to add code like https://github.com/AnacondaRecipes/aggregate/blob/master/clang/activate-clang.sh#L85-L95 and https://github.com/AnacondaRecipes/aggregate/blob/master/clang/activate-clang.sh#L110-L111

to https://github.com/AnacondaRecipes/aggregate/blob/master/gfortran-feedstock/recipe/activate-gfortran.sh

We're damned if we do, and we're damned if we don't.

[rainwoodman]

Sometimes 'pureness' shall give way for 'it just works'. Otherwise we'd just revert to the old days before anaconda ... The way anaconda builds a package is already sort of peculiar -- it has to scan the binaries and modify the files anyways. The rpath flag can be considered as part of the peculiarity -- I hope this eases some of the hard feelings.

[QM-Fitz]

I find this is helpful, I have solve the problem http://wusun.name/blog/2018-04-27-conda-libgfortran-issue/
"The fix is simple. Switch the default gfortran back to the Anaconda one. Check the path of DYLD_FALLBACK_LIBRARY_PATH. If it is empty, set it to the Anaconda lib directory, e.g., something like /Users/<your_name>/anaconda/lib. That's it. You may want to add it to the .bash_profile because macOS seems to constantly clear it. (Though I suspect that there may be a more elegant solution, at the moment this solved my problem.)" I add "export DYLD_FALLBACK_LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH:/Users//miniconda3/lib" to .bash_profile and then source ~/.bash_profile .