Failure to build julia-1.8.0-beta1 on macOS

[fxcoudert]

Trying to build the 1.8.0 beta1 on macOS as part of Homebrew testing, we're seeing a build failure. The build command is:

make VERBOSE=1 USE_BINARYBUILDER=0 prefix=/opt/homebrew/Cellar/julia/1.8.0-beta1 sysconfdir=/opt/homebrew/etc USE_SYSTEM_CSL=1 USE_SYSTEM_LIBUNWIND=1 USE_SYSTEM_PCRE=1 USE_SYSTEM_OPENLIBM=1 USE_SYSTEM_BLAS=1 USE_SYSTEM_LAPACK=1 USE_SYSTEM_GMP=1 USE_SYSTEM_MPFR=1 USE_SYSTEM_LIBSUITESPARSE=1 USE_SYSTEM_UTF8PROC=1 USE_SYSTEM_MBEDTLS=1 USE_SYSTEM_LIBSSH2=1 USE_SYSTEM_NGHTTP2=1 USE_SYSTEM_CURL=1 USE_SYSTEM_LIBGIT2=1 USE_SYSTEM_PATCHELF=1 USE_SYSTEM_ZLIB=1 USE_SYSTEM_P7ZIP=1 LIBBLAS=-lopenblas LIBBLASNAME=libopenblas LIBLAPACK=-lopenblas LIBLAPACKNAME=libopenblas USE_BLAS64=0 PYTHON=python3 MACOSX_VERSION_MIN=12 JULIA_CPU_TARGET='generic;cortex-a57;thunderx2t99;armv8.2-a,crypto,fullfp16,lse,rdm' TAGGED_RELEASE_BANNER='Built by Homebrew (v1.8.0-beta1)' install

and the failure is:

/usr/bin/tar -xmUzf /private/tmp/julia-20220312-40562-1hpnzsy/julia-1.8.0-beta1/usr-staging/zlib-cacf7f1d4e3d44d871b605da3b647f07d718623f.tgz -C /private/tmp/julia-20220312-40562-1hpnzsy/julia-1.8.0-beta1/usr
install_name_tool -id @rpath/libz.dylib /private/tmp/julia-20220312-40562-1hpnzsy/julia-1.8.0-beta1/usr/lib/libz.dylib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /private/tmp/julia-20220312-40562-1hpnzsy/julia-1.8.0-beta1/usr/lib/libz.dylib (No such file or directory)
make[2]: *** [/private/tmp/julia-20220312-40562-1hpnzsy/julia-1.8.0-beta1/usr/manifest/zlib] Error 1
make[1]: *** [julia-deps] Error 2
make: *** [install] Error 2

Full build output available there: https://gist.github.com/fxcoudert/4c0dc2b0a383dffb8e88f6286a895fd2

It seems to be building zlib (and then not finding it), while we requested USE_SYSTEM_ZLIB=1

[Bo98]

Looks like it's caused by using the custom LLVM:

julia/deps/llvm.mk

Line 89 in 7b711ce

LLVM_CMAKE += -DLLVM_ENABLE_ZLIB=ON -DZLIB_LIBRARY="$(build_prefix)/lib"

julia/deps/llvm.mk

Line 239 in 7b711ce

$(LLVM_BUILDDIR_withtype)/build-configured: | $(LLVM_PATCH_PREV) $(build_prefix)/manifest/zlib

The issue doesn't happen with USE_SYSTEM_LLVM=1.

[fxcoudert]

The issue happens whether USE_SYSTEM_ZLIB=1 is passed (as in my original report) or not.

[fxcoudert]

This will be blocking julia 1.8.0 release as part of Homebrew. Can someone please help?

[giordano]

I can reproduce the error with

make -f deps/Makefile install-zlib USE_BINARYBUILDER_ZLIB=0

My understanding is that the culprit is DESTDIR="$2" in

julia/deps/tools/common.mk

Line 132 in 9320fba

MAKE_INSTALL = +$$(MAKE) -C $1 install $$(MAKE_COMMON) $3 DESTDIR="$2"

@vtjnash what's the rationale of setting DESTDIR equal to the source directory? If I remove that, installation works as expected.

[giordano]

@fxcoudert can you please check whether

diff --git a/deps/llvm.mk b/deps/llvm.mk
index dbce7b23c4..143e455e1e 100644
--- a/deps/llvm.mk
+++ b/deps/llvm.mk
@@ -233,10 +233,14 @@ $$(LLVM_BUILDDIR_withtype)/build-compiled: $$(SRCCACHE)/$$(LLVM_SRC_DIR)/$1.patc
 LLVM_PATCH_PREV := $$(SRCCACHE)/$$(LLVM_SRC_DIR)/$1.patch-applied
 endef
 
+ifeq ($(USE_SYSTEM_ZLIB), 0)
+$(LLVM_BUILDDIR_withtype)/build-configured: | $(build_prefix)/manifest/zlib
+endif
+
 # NOTE: LLVM 12 and 13 have their patches applied to JuliaLang/llvm-project
 
 # declare that all patches must be applied before running ./configure
-$(LLVM_BUILDDIR_withtype)/build-configured: | $(LLVM_PATCH_PREV) $(build_prefix)/manifest/zlib
+$(LLVM_BUILDDIR_withtype)/build-configured: | $(LLVM_PATCH_PREV)
 
 $(LLVM_BUILDDIR_withtype)/build-configured: $(SRCCACHE)/$(LLVM_SRC_DIR)/source-extracted | $(llvm_python_workaround)
 	mkdir -p $(dir $@)

fixes for you the fact that zlib is built anyway when using USE_SYSTEM_ZLIB=1? I can't test that code path myself. I suspect this was an oversight in #42524

[vtjnash]

The value there should be derived as $$(build_staging)/$2, if you look at the comment very closely about how that macro is called.

[fxcoudert]

Launching a new build of 1.8.0-beta3 with this fix added: Homebrew/homebrew-core#100404

[fxcoudert]

While this was fixed for the case where USE_SYSTEM_ZLIB=1 is passed, the error still exists when zlib is built from source (i.e., USE_SYSTEM_ZLIB=1 is not passed).

On a clean build of master with unwind patches restored (#45189):

make USE_BINARYBUILDER=0 VERBOSE=1 CFLAGS='-Wno-implicit-function-declaration'

I get:

Install the project...
/opt/homebrew/Cellar/cmake/3.23.1/bin/cmake -P cmake_install.cmake
-- Install configuration: "Release"
-- Installing: /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389/tmp/julia/usr/lib/libz.1.2.12.dylib
-- Installing: /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389/tmp/julia/usr/lib/libz.1.dylib
-- Installing: /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389/tmp/julia/usr/lib/libz.dylib
-- Installing: /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389/tmp/julia/usr/lib/libz.a
-- Installing: /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389/tmp/julia/usr/include/zconf.h
-- Installing: /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389/tmp/julia/usr/include/zlib.h
-- Installing: /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389/tmp/julia/usr/share/man/man3/zlib.3
-- Installing: /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389/tmp/julia/usr/share/pkgconfig/zlib.pc
cd /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389/private/tmp/julia/usr && /usr/bin/tar -czf /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389.tgz.tmp .
rm -rf /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389
mv /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389.tgz.tmp /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389.tgz
[ ! -e /private/tmp/julia/usr/manifest/zlib ] || /Applications/Xcode.app/Contents/Developer/usr/bin/make uninstall-zlib
/usr/bin/tar -xmUzf /private/tmp/julia/usr-staging/zlib-21767c654d31d2dccdde4330529775c6c5fd5389.tgz -C /private/tmp/julia/usr
install_name_tool -id @rpath/libz.dylib /private/tmp/julia/usr/lib/libz.dylib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /private/tmp/julia/usr/lib/libz.dylib (No such file or directory)
make[1]: *** [/private/tmp/julia/usr/manifest/zlib] Error 1
make: *** [julia-deps] Error 2

Could you please reopen?

[giordano]

The value there should be derived as $$(build_staging)/$2, if you look at the comment very closely about how that macro is called.

@vtjnash I'm not really sure how this helps me. The call to staged-install in zlib.mk is

julia/deps/zlib.mk

Lines 16 to 19 in f18324c

	$(eval $(call staged-install, \
	zlib,$(ZLIB_SRC_DIR), \
	MAKE_INSTALL,,, \
	$(INSTALL_NAME_CMD)libz.$(SHLIB_EXT) $(build_shlibdir)/libz.$(SHLIB_EXT)))

and $(ZLIB_SRC_DIR) is correctly the path within $(BUILDDIR) where the build happens.

The problem is that doing make install ... DESTDIR=$(BUILDDIR)/$(ZLIB_SRC_DIR) in

julia/deps/tools/common.mk

Line 132 in 9320fba

MAKE_INSTALL = +$$(MAKE) -C $1 install $$(MAKE_COMMON) $3 DESTDIR="$2"

makes CMake install the library in the build directory, instead of the actual destination directory. I'm having a hard time understanding how that DESTDIR helps anything.

[vtjnash]

Oh, we may not be able to deal with symlinks in the build path very well, as that goes untested

[giordano]

My point is that I don't understand what's the purpose of DESTDIR="$2".

When building packages with autotools, the prefix is already configured at

julia/deps/tools/common.mk

Line 7 in 9320fba

CONFIGURE_COMMON = --prefix=$(abspath $(build_prefix)) --build=$(BUILD_MACHINE) --libdir=$(abspath $(build_libdir)) --bindir=$(abspath $(build_depsbindir)) $(CUSTOM_LD_LIBRARY_PATH)

and for CMake at

julia/deps/tools/common.mk

Line 21 in 9320fba

CMAKE_COMMON := -DCMAKE_INSTALL_PREFIX:PATH=$(build_prefix) -DCMAKE_PREFIX_PATH=$(build_prefix)

My understanding is that DESTDIR in make install DESTDIR=... is ignored when using autotools, but not CMake, which would override the setting of CMAKE_INSTALL_PREFIX, and the DESTDIR we supply is wrong! Do we build packages which use different build systems which require DESTDIR in the make install invocation?

@fxcoudert maybe you can see if this patch

diff --git a/deps/tools/common.mk b/deps/tools/common.mk
index 006d3486fc..acc7a55327 100644
--- a/deps/tools/common.mk
+++ b/deps/tools/common.mk
@@ -129,7 +129,7 @@ upper = $(shell echo $1 | tr a-z A-Z)
 # this rule ensures that make install is more nearly atomic
 # so it's harder to get half-installed (or half-reinstalled) dependencies
 # # and enables sharing deps compiles, uninstall, and fast reinstall
-MAKE_INSTALL = +$$(MAKE) -C $1 install $$(MAKE_COMMON) $3 DESTDIR="$2"
+MAKE_INSTALL = +$$(MAKE) -C $1 install $$(MAKE_COMMON) $3
 
 define SHLIBFILE_INSTALL
 	mkdir -p $2/$$(build_shlibdir)

wreaks havoc or solves our problems? In #44584 (comment) I reported that at least for the specific case of zlib removing DESTDIR would help.

[fxcoudert]

My understanding is that DESTDIR in make install DESTDIR=... is ignored when using autotools, but not CMake, which would override the setting of CMAKE_INSTALL_PREFIX

The meaning of DESTDIR, in autotools, is to be added to the --prefix configured at installation. So if I run ./configure --prefix=/usr/bin and then make install DESTDIR=/opt, the files will be installed into /opt/usr/bin. This page seems to state that it has the same meaning in cmake: https://cmake.org/cmake/help/latest/envvar/DESTDIR.html

[ViralBShah]

Please reopen if still an issue - and we can update the title accordingly as well.