use Tar.jl for cross platform import_docker_image

[visr]

Now that Tar.jl is released we can try out the fix proposed in #520.
Fixes #520.

[codecov]

Codecov Report

Merging #548 into master will decrease coverage by 3%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master     #548      +/-   ##
==========================================
- Coverage   57.09%   54.09%   -3.01%     
==========================================
  Files          28       28              
  Lines        2804     2795       -9     
==========================================
- Hits         1601     1512      -89     
- Misses       1203     1283      +80

Impacted Files	Coverage Δ
src/BinaryBuilder.jl	43.07% <ø> (-1.54%)	⬇️
src/DockerRunner.jl	0% <0%> (-88.24%)	⬇️
src/Rootfs.jl	52.19% <0%> (-3.85%)	⬇️
src/OutputCollector.jl	90.43% <0%> (-2.61%)	⬇️
src/squashfs_utils.jl	79.48% <0%> (-2.57%)	⬇️
src/Runner.jl	93.03% <0%> (-1%)	⬇️
src/UserNSRunner.jl	77.12% <0%> (+0.81%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6a8cdb4...6846d89. Read the comment docs.

[visr]

Ok this seems to be getting me a step closer to running a BinaryBuilder shell on Windows.

Previously in #441 (comment) it crashed on not being able to find docker_entrypoint.sh. With this PR it crashes on permission denied to run docker_entrypoint.sh. Progress!

julia> using BinaryBuilder; BinaryBuilder.runshell(Linux(:x86_64, libc=:glibc), verbose=true)
[ Info: Checking to see if c:\Users\visser_mn\.julia\dev\BinaryBuilder\ is encrypted...
[ Info: Couldn't open /proc/mounts, returning...
[ Info: Checking to see if c:\Users\visser_mn\.julia\dev\BinaryBuilder\deps\ is encrypted...
[ Info: Couldn't open /proc/mounts, returning...
[ Info: Importing docker base image from C:\Users\visser_mn\.julia\artifacts\0f1867e8dbf4dfeaa77bb52ad707b86f74f0c80e to julia_binarybuilder_rootfs:v2019.11.22-f0c80e
docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"/docker_entrypoint.sh\": permission denied": unknown.
ERROR: failed process: Process(`docker run --rm --privileged -w '/\workspace\' -v 'C:\Users\visser_mn\.julia\artifacts\5880b437d037f940210708d848003d8a96ab37b6:/opt/x86_64-linux-gnu/PlatformSupport-2019.11.11:ro' -v 'C:\Users\visser_mn\.julia\artifacts\f2124900465e4fa506f8b1e274074eb92d68d46b:/opt/x86_64-linux-gnu/GCCBootstrap-4.8.5:ro' -v 'C:\Users\visser_mn\.julia\artifacts\4a2898dd43f5f7141ff186a5c671abd79a91355c:/opt/x86_64-linux-musl/LLVMBootstrap-8.0.0:ro' -v 'C:\Users\visser_mn\.julia\artifacts\1671292e86c31ca56ac8a9a37b1fb034ce9ff7f6:/opt/x86_64-linux-musl/PlatformSupport-2019.11.11:ro' -v 'C:\Users\visser_mn\.julia\artifacts\2e6f62648ec0497cc25774aa175471ad41b6172a:/opt/x86_64-linux-musl/GCCBootstrap-4.8.5:ro' -v 'C:\Users\visser_mn\.julia\dev\BinaryBuilder:/workspace' -v 'C:\Users\visser_mn\AppData\Local\Temp\jl_SR2zIm:/opt/bin' -e HOSTFC=x86_64-linux-musl-gfortran -e HOSTSTRIP=x86_64-linux-musl-strip -e OBJCOPY_BUILD=x86_64-linux-musl-objcopy -e HISTFILE=/meta/.bash_history -e HOSTRANLIB=x86_64-linux-musl-ranlib -e VERBOSE=true -e BUILD_STRIP=x86_64-linux-musl-strip -e RUSTUP_TOOLCHAIN=stable-x86_64-unknown-linux-gnu -e LLVM_TARGET=x86_64-linux-gnu -e BUILD_AS=x86_64-linux-musl-as -e RUSTC=rustc -e BUILD_READELF=x86_64-linux-musl-readelf -e LD_FOR_BUILD=x86_64-linux-musl-ld -e NM_FOR_BUILD=x86_64-linux-musl-nm -e HOSTCC=x86_64-linux-musl-gcc -e BUILD_CC=x86_64-linux-musl-gcc -e CXX_BUILD=x86_64-linux-musl-g++ -e RUSTUP_HOME=/opt/x86_64-linux-gnu -e HOSTREADELF=x86_64-linux-musl-readelf -e OBJC=objc -e prefix=/workspace/destdir -e READELF_FOR_BUILD=x86_64-linux-musl-readelf -e OBJDUMP_BUILD=x86_64-linux-musl-objdump -e AS_BUILD=x86_64-linux-musl-as -e CMAKE_TARGET_TOOLCHAIN=/opt/x86_64-linux-gnu/x86_64-linux-gnu.cmake -e PKG_CONFIG_PATH=/workspace/destdir/lib/pkgconfig:/workspace/destdir/share/pkgconfig -e AR_FOR_BUILD=x86_64-linux-musl-ar -e FC_FOR_BUILD=x86_64-linux-musl-gfortran -e nproc=8 -e CC=cc -e target=x86_64-linux-gnu -e HOSTOBJDUMP=x86_64-linux-musl-objdump -e LD_BUILD=x86_64-linux-musl-ld -e HOSTCXX=x86_64-linux-musl-g++ -e bindir=/workspace/destdir/bin -e CCACHE_COMPILERCHECK=content -e HOSTNM=x86_64-linux-musl-nm -e AR_BUILD=x86_64-linux-musl-ar -e LIPO_BUILD=x86_64-linux-musl-lipo -e nbits=64 -e MACHTYPE=x86_64-linux-musl -e RANLIB_BUILD=x86_64-linux-musl-ranlib -e CARGO_BUILD_TARGET=x86_64-unknown-linux-gnu -e CC_BUILD=x86_64-linux-musl-gcc -e DSYMUTIL_BUILD=llvm-dsymutil -e OBJDUMP_FOR_BUILD=x86_64-linux-musl-objdump -e ZERO_AR_DATE=1 -e CXX=c++ -e LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/usr/local/lib:/usr/lib:/lib64:/lib:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib64:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib64:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib:/workspace/destdir/lib64:/workspace/destdir/lib -e HOSTOBJCOPY=x86_64-linux-musl-objcopy -e HOSTLIPO=x86_64-linux-musl-lipo -e BUILD_RANLIB=x86_64-linux-musl-ranlib -e STRIP_BUILD=x86_64-linux-musl-strip -e HOSTAR=x86_64-linux-musl-ar -e USER=julia -e GOPATH=/workspace/.gopath -e GO=go -e rust_host=x86_64-unknown-linux-gnu -e BUILD_OBJDUMP=x86_64-linux-musl-objdump -e exeext= -e SOURCE_DATE_EPOCH=0 -e NM_BUILD=x86_64-linux-musl-nm -e CXX_FOR_BUILD=x86_64-linux-musl-g++ -e BUILD_OBJCOPY=x86_64-linux-musl-objcopy -e rust_target=x86_64-unknown-linux-gnu -e USE_CCACHE=false -e FC=gfortran -e HOSTDSYMUTIL=llvm-dsymutil -e BUILD_FC=x86_64-linux-musl-gfortran -e CC_FOR_BUILD=x86_64-linux-musl-gcc -e GOARM=7 -e CARGO_HOME=/opt/x86_64-linux-gnu -e CARGO=cargo -e dlext=so -e SRC_NAME= -e PATH=/opt/bin:/opt/x86_64-linux-gnu/bin:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/destdir/bin -e FC_BUILD=x86_64-linux-musl-gfortran -e AS_FOR_BUILD=x86_64-linux-musl-as -e BUILD_AR=x86_64-linux-musl-ar -e MESON_HOST_TOOLCHAIN=/opt/x86_64-linux-musl/x86_64-linux-musl.meson -e BUILD_CXX=x86_64-linux-musl-g++ -e CMAKE_HOST_TOOLCHAIN=/opt/x86_64-linux-musl/x86_64-linux-musl.cmake -e proc_family=intel -e BUILD_LD=x86_64-linux-musl-ld -e V=true -e STRIP_FOR_BUILD=x86_64-linux-musl-strip -e PKG_CONFIG_SYSROOT_DIR=/workspace/destdir -e TERM=screen -e MESON_TARGET_TOOLCHAIN=/opt/x86_64-linux-gnu/x86_64-linux-gnu.meson -e OBJCOPY_FOR_BUILD=x86_64-linux-musl-objcopy -e RANLIB_FOR_BUILD=x86_64-linux-musl-ranlib -e BUILD_LIPO=x86_64-linux-musl-lipo -e GOCACHE=/workspace/.gocache -e HOSTLD=x86_64-linux-musl-ld -e 'HIDDEN_PS1=\[\]sandbox\[\]:\[\]${PWD//$WORKSPACE/$\{WORKSPACE\}}\[\] \$ ' -e DSYMUTIL_FOR_BUILD=llvm-dsymutil -e BUILD_DSYMUTIL=llvm-dsymutil -e LIPO_FOR_BUILD=x86_64-linux-musl-lipo -e LLVM_HOST_TARGET=x86_64-linux-musl -e READELF_BUILD=x86_64-linux-musl-readelf -e libdir=/workspace/destdir/lib -e BUILD_NM=x86_64-linux-musl-nm -e bb_target=x86_64-linux-gnu -e WORKSPACE=/workspace -e HOSTAS=x86_64-linux-musl-as -ti -i julia_binarybuilder_rootfs:v2019.11.22-f0c80e /bin/bash -l`, ProcessExited(126)) [126]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] #run#565(::Bool, ::typeof(run), ::Cmd) at .\process.jl:440
 [3] run at .\process.jl:438 [inlined]
 [4] #run_interactive#283(::Nothing, ::Nothing, ::Nothing, ::Bool, ::typeof(BinaryBuilder.run_interactive), ::BinaryBuilder.DockerRunner, ::Cmd) at c:\Users\visser_mn\.julia\dev\BinaryBuilder\src\DockerRunner.jl:207
 [5] #run_interactive at .\none:0 [inlined]
 [6] #runshell#211 at c:\Users\visser_mn\.julia\dev\BinaryBuilder\src\Runner.jl:613 [inlined]
 [7] #runshell at .\none:0 [inlined]
 [8] #runshell#212(::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(BinaryBuilder.runshell), ::Type{BinaryBuilder.DockerRunner}, ::Linux) at c:\Users\visser_mn\.julia\dev\BinaryBuilder\src\Runner.jl:617
 [9] (::BinaryBuilder.var"#kw##runshell")(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(BinaryBuilder.runshell), ::Type{BinaryBuilder.DockerRunner}, ::Linux) at .\none:0
 [10] #runshell#210(::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:verbose,),Tuple{Bool}}}, ::typeof(BinaryBuilder.runshell), ::Linux) at c:\Users\visser_mn\.julia\dev\BinaryBuilder\src\Runner.jl:609
 [11] (::BinaryBuilder.var"#kw##runshell")(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(BinaryBuilder.runshell), ::Linux) at .\none:0
 [12] top-level scope at REPL[2]:1

Makes sense, running this command and then checking docker images previously showed that 0B julia_binarybuilder_rootfs images were being created, and now it is 271MB.

[staticfloat]

Fascinating. I'm betting what's happening is that Tar.jl doesn't see the executable bit being set on windows. This line in particular:

https://github.com/JuliaLang/Tar.jl/blob/master/src/create.jl#L25

This will always be marked as non-executable because we don't get the right mode on Windows: JuliaLang/julia#33212

[visr]

I see this code lives in BinaryBuilderBase now, so I'll close this PR.

I looked into submitting this there, I see Tar is already a dependency, but looking at this comment:
https://github.com/JuliaPackaging/BinaryBuilderBase.jl/blob/b33bde58330f7bea87f150de444961758be685e8/src/Prefix.jl#L274-L287

It seems that it may instead be more desirable to use tar and unzip JLL's instead. Tar exists, unzip not yet.