Switch to conda-built gfortran - also for osx-64 - and do bundling in this repo?

[h-vetinari]

In the discussion in MacPython/scipy-wheels#140, @isuruf suggested to reuse the conda-forge gfortran builds, which are already being used for osx-arm here, and Ralf agreed that this would be fine.

Currently the arm binaries are published on @isuruf's fork but come with complete instructions for reproducing them. In order not to place this burden on him (to bundle various versions for osx-x64 or newer gcc versions), we should execute these instructions somewhere - sounds like it would make sense in this repo...

PS. I thought it would be nice to be able to jump to using the same gfortran version for osx-x64 & osx-arm64, so I opened a PR (still WIP).

PPS. Scipy is fine to bump the minimum OSX-requirement to 10.13 (since this is what cibuildwheel now sets as minimum. apparently), though AFAICT, conda-forge still builds for a minimum of 10.9.

[h-vetinari]

@isuruf
You asked me to open this issue and I plan to tackle this - any preferences where/how it should be done?

My guess is that we don't want to do it within gfortran_utils.sh, because that get's called from the wheel repo, and we don't want to have conda interfering there.

So the biggest question (to me) is where we set up conda, and execute the script to bundle gfortran.

[h-vetinari]

@mattip @matthew-brett
Do you have an opinion on where a potential bundling of gfortran built by conda-forge should go (in terms of code & execution)? I could imagine doing it within the CI of this repo, and then pushing the artefacts to something like https://anaconda.org/multibuild-wheels-staging...

[mattip]

in the archives folder here?

[h-vetinari]

in the archives folder here?

Do you mean the final artefacts? That sounds OK to me of course, though I'm mostly wondering about where to put the following code (taken from https://github.com/isuruf/gcc/releases), and most importantly where/how to execute it.

ver=10.2.1
conda create -n gfortran-darwin-arm64 gfortran_impl_osx-arm64=${ver} libgfortran-devel_osx-arm64=${ver} --yes
CONDA_SUBDIR=osx-arm64 conda install -n gfortran-darwin-arm64 libgfortran5=${ver} --yes
conda activate gfortran-darwin-arm64
rm -rf $CONDA_PREFIX/lib/{libc++*,*.a,pkgconfig,clang}
rm -rf $CONDA_PREFIX/{include,conda-meta,bin/iconv}
for f in $CONDA_PREFIX/lib/{libgmp.dylib,libgmpxx.dylib,libisl.dylib,libiconv.dylib,libmpfr.dylib,libz.dylib,libcharset.dylib,libmpc.dylib}; do
  install_name_tool -delete_rpath $CONDA_PREFIX/lib $f || true;
  install_name_tool -delete_rpath $CONDA_PREFIX/lib $f || true;
  rm $f;
done
rm $CONDA_PREFIX/lib/libiomp5.dylib
mv $CONDA_PREFIX/lib/{libgfortran*,libgomp*,libomp*,libgcc_s*} $CONDA_PREFIX/lib/gcc/arm64-apple-darwin20.0.0/${ver}
for f in libgfortran.dylib libgfortran.5.dylib libgcc_s.2.dylib; do
  install_name_tool -delete_rpath $CONDA_PREFIX/lib $CONDA_PREFIX/lib/gcc/arm64-apple-darwin20.0.0/${ver}/$f || true;
  install_name_tool -delete_rpath $CONDA_PREFIX/lib $CONDA_PREFIX/lib/gcc/arm64-apple-darwin20.0.0/${ver}/$f || true;
done
ln -sf /usr/bin/ld $CONDA_PREFIX/libexec/gcc/arm64-apple-darwin20.0.0/${ver}/ld
sed -i '' "s#-rpath $CONDA_PREFIX/lib##g" $CONDA_PREFIX/lib/gcc/arm64-apple-darwin20.0.0/${ver}/libgfortran.spec
cd $CONDA_PREFIX/../
grep -ir "isuruf" gfortran-darwin-arm64/
tar -czf ../../gfortran-darwin-arm64.tar.gz gfortran-darwin-arm64

[mattip]

"how to execute it" - it seems to need an aarch64 (arm-on-M1) machine. I don't think those can be emulated yet. So someone could execute the code (which could be stored in this repo with directions for replication) on a private machine and upload the artifact. Not so good for reproducibility.

[h-vetinari]

"how to execute it" - it seems to need an aarch64 (arm-on-M1) machine.

Pretty sure there would be ways to make that work. conda-lock does it too, for example.

So someone could execute the code (which could be stored in this repo with directions for replication) on a private machine and upload the artifact. Not so good for reproducibility.

I think it'd be a big win if we can take the dependency on individual members of out the equation. :)

[isuruf]

It seems to need an aarch64 (arm-on-M1) machine.

Nope. the compiler is a cross compiler that generates code for aarch64, but runs on x86_64.

[mattip]

Done in #11

[h-vetinari]

Done in #11

Well, partially. This issue was mainly concerned about also finding a home for the build script that Isuru used to extract the artefact from conda-forge.

[mattip]

conda is installed by default in the macos-11 images (Miniconda 4.12.0).

[h-vetinari]

conda is installed by default in the macos-11 images (Miniconda 4.12.0).

OK cool, that makes things a bit easier. We still need to decide a place where to run the script though. Do we want to combine the extraction from conda-forge with the installation? If not, do we have a two-stage build here, or build gfortran-darwin-${arch}-${type}.tar.gz somewhere else?

[mattip]

I think it is fine to commit the build script here for reproducibility. Where the actual artifact is hosted is less of an issue, the current solution of hosting it at https://github.com/isuruf/gcc/releases is fine as long as we can verify and track it.

[h-vetinari]

I think it is fine to commit the build script here for reproducibility. Where the actual artifact is hosted is less of an issue, the current solution of hosting it at https://github.com/isuruf/gcc/releases is fine as long as we can verify and track it.

I don't think it's a good solution, as it's non-automated and depends on the access of one person. The point is to build this somewhere so that people from numpy/scipy can rerun the build-script at will.

We have two steps:

1. build gfortran-darwin-${arch}-${type}.tar.gz from the conda-package
2. install it (this repo)

We could combine 1. & 2. in this repo. If we want to persist the artefact from 1., then I'm proposing we also build it in this repo, and store it wherever we store the other artefacts (anaconda nightly?).

[mattip]

The artifact is also downloaded as part of numpy/numpy testing and in wheel building to pull out the libgfortran.5.dylib, so if it moves, those need updating.