Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use make install for linux build #129

Merged
merged 9 commits into from
Oct 11, 2021
Merged

Conversation

straight-shoota
Copy link
Member

@straight-shoota straight-shoota commented Sep 21, 2021

Since crystal-lang/crystal#10878 was merged, we can replace the custom installation steps with make install. shards did already have make install for a long time and we can use that as well.

There are a couple of differences in the resulting tarball:

--- <(tar -tf crystal-1.2.0-dev-1-linux-x86_64.original.tar)
+++ <(tar -tf crystal-1.2.0-dev-1-linux-x86_64.new.tar)
@@ -3,67 +3,19 @@
 crystal-1.2.0-dev-1/lib/crystal/
 crystal-1.2.0-dev-1/lib/crystal/lib/
 crystal-1.2.0-dev-1/lib/crystal/lib/libgc.a
-crystal-1.2.0-dev-1/lib/crystal/bin/
-crystal-1.2.0-dev-1/lib/crystal/bin/shards
-crystal-1.2.0-dev-1/lib/crystal/bin/crystal
 crystal-1.2.0-dev-1/share/
 crystal-1.2.0-dev-1/share/man/
 crystal-1.2.0-dev-1/share/man/man1/
-crystal-1.2.0-dev-1/share/man/man1/shards.1.gz
+crystal-1.2.0-dev-1/share/man/man1/shards.1
 crystal-1.2.0-dev-1/share/man/man1/crystal.1.gz
 crystal-1.2.0-dev-1/share/man/man5/
-crystal-1.2.0-dev-1/share/man/man5/shard.yml.5.gz
-crystal-1.2.0-dev-1/share/doc/
-crystal-1.2.0-dev-1/share/doc/crystal/
-crystal-1.2.0-dev-1/share/doc/crystal/examples/
-crystal-1.2.0-dev-1/share/doc/crystal/examples/spectral-norm.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/red_black_tree.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/wordcount.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/binary-trees.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/llvm/
-crystal-1.2.0-dev-1/share/doc/crystal/examples/llvm/brainfuck.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/meteor.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/http_server.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/tree.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/channel_select.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/impl.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sudoku.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sieve.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/neural_net.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/2048.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/noise.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/nbodies.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/conway.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/pretty_json.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/quine.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/matmul.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/brainfuck.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/degree_days.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/text_raytracer.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/fannkuch-redux.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/egrep.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/mt_gc_test.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/channel_primes.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/pig.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sdl/
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sdl/raytracer.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sdl/fire.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sdl/fire.txt
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sdl/tv.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sdl/tv.txt
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sdl/sdl/
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sdl/sdl/lib_sdl.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sdl/sdl/surface.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/sdl/sdl/sdl.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/mandelbrot.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/mandelbrot2.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/tcp_client.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/havlak.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/compiler/
-crystal-1.2.0-dev-1/share/doc/crystal/examples/compiler/visitor_example.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/compiler/formatter_example.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/compiler/transformer_example.cr
-crystal-1.2.0-dev-1/share/doc/crystal/examples/tcp_server.cr
+crystal-1.2.0-dev-1/share/man/man5/shard.yml.5
+crystal-1.2.0-dev-1/share/zsh/
+crystal-1.2.0-dev-1/share/zsh/site-functions/
+crystal-1.2.0-dev-1/share/zsh/site-functions/_crystal
+crystal-1.2.0-dev-1/share/bash-completion/
+crystal-1.2.0-dev-1/share/bash-completion/completions/
+crystal-1.2.0-dev-1/share/bash-completion/completions/crystal
 crystal-1.2.0-dev-1/share/licenses/
 crystal-1.2.0-dev-1/share/licenses/crystal/
 crystal-1.2.0-dev-1/share/licenses/crystal/LICENSE
@@ -130,7 +82,6 @@
 crystal-1.2.0-dev-1/share/crystal/src/llvm/ext/
 crystal-1.2.0-dev-1/share/crystal/src/llvm/ext/find-llvm-config
 crystal-1.2.0-dev-1/share/crystal/src/llvm/ext/llvm-versions.txt
-crystal-1.2.0-dev-1/share/crystal/src/llvm/ext/llvm_ext.o
 crystal-1.2.0-dev-1/share/crystal/src/llvm/ext/llvm_ext.cc
 crystal-1.2.0-dev-1/share/crystal/src/llvm/value_methods.cr
 crystal-1.2.0-dev-1/share/crystal/src/llvm/function_collection.cr
  • The binaries are placed directly in bin/ instead of lib/crystal/bin. Since Add CrystalPath.expand_paths, expand relative to compiler path crystal#11030 there's no need for a wrapper script anymore. We could consider to still keep using it, but I don't think that's necessary. shards never required a wrapper script and bin/shards was just a symlink to lib/crystal/bin/shards.
  • shards manpages are uncompressed (should be fixed with Compress manpages on install shards#524).
  • The entire examples folder has been removed. This was discussed as part of the make install recipe (Add make install crystal#10878 (review)). Examples should be considered part of the documentation (or a separate samples package).
  • Shell completion scripts have been added. They were just plainly missing from the tarball before. This is already a positive effect of make install.
  • llvm_ext.o has been removed. This object file must not be part of the binary distribution package. It depends on the libraries of the target system and is not generically usable. See Move llvm_ext.o to .build crystal#10880

The omnibus script could also make use of make install, but I don't have a mac to test that locally. So this is only for the linux tarball build.

Resolves #38

@bcardiff
Copy link
Member

Without the .sh wrapper how would the .tar.gz work at any path where it is expanded?

It's fine to remove the use of the wrapper in distro packages but having the .tar.gz work with just expansion (and having all the required libs) is ver comfortable.

@straight-shoota
Copy link
Member Author

straight-shoota commented Sep 21, 2021

By now, the binary should work at any location without the wrapper.

What feature does the wrapper still provide that the binary doesn't after crystal-lang/crystal#11030 ?

The tar.gz has never contained all required libraries (that's added with #102 but it will be a separate tar.gz). It only ships with our custom build of libgc. That still works and it's linked correctly (the latest commit fixes the lib path configuration).

@bcardiff
Copy link
Member

What feature does the wrapper still provide that the binary doesn't after crystal-lang/crystal#11030 ?

I was missing that point. Wrapper scripts is no longer needed 👍

The tar.gz has never contained all required libraries

That's correct. (I think) I didn't imply the opposite.


If the shipped compiler was built with CRYSTAL_CONFIG_LIBRARY_PATH=$ORIGIN/../lib/crystal, how is the the next gen going to grab the gc built in this docker file. I think it will grab the one embedded in the previous compiler always. I think we will be now missing setting CRYSTAL_LIBRARY_PATH to a non empty value in order to prevent the libs from $ORIGIN/../lib/crystal getting to the next gen.

@straight-shoota
Copy link
Member Author

(I think) I didn't imply the opposite.

👍 I was a little confused by "having all the required libs", but I assumed you meant essentially libgc by that (currently). Just wanted to record that we don't really package all required libs, in case somebody would misinterpret that.

Good point about CRYSTAL_LIBRARY_PATH. Yeah, we might need to tweak the Makfile for that. Or the bin/crystal script from the compiler repo?

@bcardiff
Copy link
Member

Having another makefile option to ignore_embedded_libs is fine probably. The implementation of that will still be to use a non existing path to override the default option that includes origin.

Whatever we choose should be announced for maintainers since they need to do something explicitly in order to avoid linking to the embedded libs.

Maybe it would be better to ignore by default the embedded libs in the compiler itself? 🤔

@straight-shoota
Copy link
Member Author

For the record, any issues with library paths are a direct effect of crystal-lang/crystal#11030 and not directly related to make install itself. But 9223ef8 is still necessary because the location of the lib directory changes with the default baked-in CRYSTAL_LIBRARY_PATH.

Technically, we could consider resolving the lib path issue first and then switching to make install.

As far as I can see, the only thing we need to change is this:

--- c/linux/Dockerfile
+++ i/linux/Dockerfile
@@ -59,7 +59,7 @@ RUN git clone https://github.com/ivmai/bdwgc \
  && ./configure --disable-debug --disable-shared --enable-large-config \
  && make -j$(nproc) CFLAGS=-DNO_GETCONTEXT

-ENV LIBRARY_PATH=/bdwgc/.libs/
+ENV CRYSTAL_LIBRARY_PATH=/bdwgc/.libs/
 RUN llvm-config --version

 ARG previous_crystal_release

This explicitly overrides the baked-in path from CRYSTAL_CONFIG_LIBRARY_PATH.

Another path-related issue is that the deep-path to the binary is added to PATH, instead of the location of the wrapper script. This deep path no longer exists if we replace the wrapper script with the binary directly.

--- c/linux/Dockerfile
+++ i/linux/Dockerfile
@@ -64,7 +64,7 @@ RUN llvm-config --version

 ARG previous_crystal_release
 ADD ${previous_crystal_release} /tmp/crystal.tar.gz
-ENV PATH=${PATH}:/tmp/crystal/lib/crystal/bin/
+ENV PATH=${PATH}:/tmp/crystal/bin/
 RUN mkdir -p /tmp/crystal \
   && tar xz -f /tmp/crystal.tar.gz -C /tmp/crystal --strip-component=1 \
   && crystal --version \

Now, we could consider leaving the old paths (lib/crystal/bin and lib/crystal/lib) as symbolic links to the new ones (bin and lib) for a transition period. This would allow using the same instructions for packages with the old and new paths.

@bcardiff
Copy link
Member

Changing CRYSTAL_LIBRARY_PATH is enough. But if the goal is to have a make install target that others can use I think it should have some parameters as a more stable API.

Regarding the PATH set in docker, having symlinks or both paths added ENV PATH=${PATH}:/tmp/crystal/lib/crystal/bin/:/tmp/crystal/bin/ should work as a transition. 👍

@straight-shoota
Copy link
Member Author

Yeah, I think symlinks is probably a better solution because it works for anyone using the tar.gz with the previous paths.

At some point, we'll have to break that, though. But at least we can offer this for a transition period.

@straight-shoota
Copy link
Member Author

With the last two commits, the build works for any bootstrap version with new or old library paths.

At some point we just need to remove the legacy symlinks and update PATH (the latter can be done as soon as we don't need any pre-1.2.0 release as bootstrap compiler).

@straight-shoota straight-shoota changed the title Use make install for linux build Use make install for linux build Oct 1, 2021
@straight-shoota
Copy link
Member Author

@straight-shoota
Copy link
Member Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Missing bash and zsh completion in linux packages
2 participants