Add static linking to rust #10528
Add static linking to rust #10528
Conversation
|
This doesn't appear to update the docs about the changes to |
|
There was actually no change to the existing In general, there appears to be very little documentation about rust's linkage model, and I would like to change that. I'm not sure where the best place for this documentation is. I would imagine it would be That being said, I'd like to verify that this is the direction that we wish to pursue before writing up that document. I'm perfectly ok with blocking this for me to write that document, however. |
|
There's tutorial-ffi, which would presumably cover the new |
|
I have hit a snag in getting
This is causing programs to crash because one runtime is initialized and the other isn't. This is a more general problem, however, as it doesn't only apply to libstd (anything using conditions would be affected). This isn't a rust-specific problem, and in my experimentation with the linker I was unable to produce a binary which had the expected behavior. Additionally, this is a subset of the larger problem of guaranteeing that there is only one copy of a library in a final product. The compiler currently has no means to generate a dynamic library or executable with the guarantee that each upstream library will appear exactly once. We currently have this invariant because everything is linked dynamically, but this is not the case with static linking. With this in mind, I do not believe that favoring static linking is the right thing to do at this time. I'm going to change this patch to require dynamic linking when creating a dynamic library, and add a I believe that we can work around this issue, but it requires more infrastructure than i believe that we possess today. I have not given this a large amount of thought, but all I can come up with is that this requires some complicated analysis which likely isn't worth it (this is "just linking" after all). I'm a little up in the air on whether to close this pull request because of this issue or not. I personally want to see this feature working in the compiler, but I'm uncomfortable with the amount of thought which I have put into this (I do not believe it is sufficiently enough). I will discuss this more and see what comes of it. |
|
I may have got your last comment wrong, but it looks like what you want is actually to do a single linking pass with mixed static/dynamic objs (libfoo.so, libstd.a), have all occurences of libstd symbols with |
|
I believe that we have reasons to not mix static/dynamic libraries right now. I can't think of a fantastic way to guarantee that you only ever have one copy of a library. As implemented, all result artifacts are required to be all-static or all-dynamic for this reason. What do you mean by a global binding? I'm not sure I've heard of how that works. |
|
I agree with your assessment. |
|
I have updated this to no longer use the I have added an appropriate test, and we no longer mangle object files at all after generation. They're simply shipped through the system in archives until the final destination link step. |
|
👍, but another set of eyeballs would be nice. |
|
@alexcrichton Do you think this is OK to review and merge? |
|
I believe it is, yes. |
|
|
||
| fn run_ar(sess: Session, args: &str, cwd: Option<&Path>, | ||
| paths: &[&Path]) -> ProcessOutput { | ||
| let ar = sess.opts.ar.clone().unwrap_or(~"ar"); |
There was a problem hiding this comment.
nit: this will be slightly faster if you don't unconditionally allocate ~"ar"
|
r=me modulo comments. |
This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes rust-lang#552
This infrastructure is meant to support runnings tests that involve various interesting interdependencies about the types of crates being linked or possibly interacting with C libraries. The goal of these make tests is to not restrict them to a particular test runner, but allow each test to run its own tests. To this end, there is a new src/test/run-make directory which has sub-folders of tests. Each test requires a `Makefile`, and running the tests constitues simply running `make` inside the directory. The new target is `check-stageN-rmake`. These tests will have the destination directory (as TMPDIR) and the local rust compiler (as RUSTC) passed along to them. There is also some helpful cross-platform utilities included in src/test/run-make/tools.mk to aid with compiling C programs and running them. The impetus for adding this new test suite is to allow various interesting forms of testing rust linkage. All of the tests initially added are various flavors of compiling Rust and C with one another as well as just making sure that rust linkage works in general. Closes rust-lang#10434
In rust-lang#10422, I didn't actually test to make sure that the '-Z gen-crate-map' option was usable before I implemented it. The crate map was indeed generated when '-Z gen-crate-map' was specified, but the I/O factory slot was empty because of an extra check in trans about filling in that location. This commit both fixes that location, and checks in a "fancy test" which does lots of fun stuff. The test will use the rustc library to compile a rust crate, and then compile a C program to link against that crate and run the C program. To my knowledge this is the first test of its kind, so it's a little ad-hoc, but it seems to get the job done. We could perhaps generalize running tests like this, but for now I think it's fine to have this sort of functionality tucked away in a test.
In this series of commits, I've implemented static linking for rust. The scheme I implemented was the same as my [mailing list post](https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html). The commits have more details to the nitty gritty of what went on. I've rebased this on top of my native mutex pull request (#10479), but I imagine that it will land before this lands, I just wanted to pre-emptively get all the rebase conflicts out of the way (becuase this is reorganizing building librustrt as well). Some contentious points I want to make sure are all good: * I've added more "compiler chooses a default" behavior than I would like, I want to make sure that this is all very clearly outlined in the code, and if not I would like to remove behavior or make it clearer. * I want to make sure that the new "fancy suite" tests are ok (using make/python instead of another rust crate) If we do indeed pursue this, I would be more than willing to write up a document describing how linking in rust works. I believe that this behavior should be very understandable, and the compiler should never hinder someone just because linking is a little fuzzy.
This registers new snapshots after the landing of #10528, and then goes on to tweak the build process to build a monolithic `rustc` binary for use in future snapshots. This mainly involved dropping the dynamic dependency on `librustllvm`, so that's now built as a static library (with a dynamically generated rust file listing LLVM dependencies). This currently doesn't actually make the snapshot any smaller (24MB => 23MB), but I noticed that the executable has 11MB of metadata so once progress is made on #10740 we should have a much smaller snapshot. There's not really a super-compelling reason to distribute just a binary because we have all the infrastructure for dealing with a directory structure, but to me it seems "more correct" that a snapshot compiler is just a `rustc` binary.
|
@alexcrichton do you have some numbers for statically linked binaries (filesize) ? |
|
@rofl0r: hello world is 2.5M. With -Z lto it's 1.7M. |
Clear with drain changelog: [`clear_with_drain`]: Add new lint Fixes rust-lang#9339
In this series of commits, I've implemented static linking for rust. The scheme I implemented was the same as my mailing list post.
The commits have more details to the nitty gritty of what went on. I've rebased this on top of my native mutex pull request (#10479), but I imagine that it will land before this lands, I just wanted to pre-emptively get all the rebase conflicts out of the way (becuase this is reorganizing building librustrt as well).
Some contentious points I want to make sure are all good:
If we do indeed pursue this, I would be more than willing to write up a document describing how linking in rust works. I believe that this behavior should be very understandable, and the compiler should never hinder someone just because linking is a little fuzzy.