Compile a crate from its source archive directly

[fenollp]

Problem

Today, crates provided by e.g. crates.io are downloaded in a (somewhat loosely defined?) archive format, with a .crate extension. Example: https://static.crates.io/crates/clap/clap-3.0.6.crate

clap-3.0.6.crate: gzip compressed data, was "clap-3.0.6.crate", max compression, original size modulo 2^32 1110528

This gets stored as a local cache and is also extracted in a nearby folder, which then gets fed to cargo / rustc for compilation.

I (and surely others) am developing a $RUSTC_WRAPPER that has to deal with these archives and thus have to replicate the handling of said .crates. Given these open issues, and for optimization reasons, I would like for cargo (or rustc) to be able to take in these archives directly, without prior extraction:

• deterministic source archives cargo#2948
• Migrate to alternative compression for crates.io crates cargo#2526
• Consider making the src cache read-only. cargo#9455

Proposed Solution

I am not familiar with the inner workings of either cargo nor rustc so this is a high level take.

Basically, I'm asking that today's call to rustc be seamlessly interchangeable from:

rustc --crate-name clap ...... $HOME/.cargo/registry/src/index.crates.io-6f17d22bba15001f/clap-3.0.6/src/lib.rs ....

to

rustc --crate-name clap ...... $HOME/.cargo/registry/cache/index.crates.io-6f17d22bba15001f/clap-3.0.6.crate ...

thus moving the crate archive handling code to within the boundary of (I guess) rustc.

This in turn

• encapsulates the archive format
• allows me (and others) to just pass archives down from crates.io straight to rustc

Notes

I realize this issue may need to move to rustc. But hey let's have a small discussion first :)

[epage]

https://internals.rust-lang.org/t/jar-for-rust-single-file-crate-support-for-rustc/19576 is the last conversation I saw for compiling a .crate without extracting it.

I (and surely others) am developing a $RUSTC_WRAPPER that has to deal with these archives and thus have to replicate the handling of said .crates

This part isn't making sense to me. If you are passing a $RUSTC_WRAPPER to cargo, why are you needing to build .crate files as cargo would have extracted it for you and told you where the source is.

allows me (and others) to just pass archives down from crates.io straight to rustc

Sounds like you aren't creating a $RUSTC_WRAPPER but creating your own build tool on top of rustc, is that right?

Then this is purely a rustc concern and not a Cargo concern, though designing this in a way for Cargo to take advantage of would be useful.

[workingjubilee]

I feel like it would be better to first librarify the relevant code in cargo and then discuss adding it to rustc?

[epage]

A lot of that depends on what rustc would want.

For example, one idea that came up in that thread is if we could abstract this in a way for cargo to define the filesystem so we could read from git databases directly, rather than checking out files. Would we want that in scope? What would it look like to support something like that?

[fenollp]

cargo would have extracted it for you
Sounds like you aren't creating a $RUSTC_WRAPPER

Well you're right, this issue should probably live in rustc's repo :)
I am indeed creating a wrapper but I am also handling fetching and extracting crates.io deps as an optimization. See e.g.

FROM scratch AS cratesio-ring-0.17.8-index.crates.io-6f17d22bba15001f
ADD --chmod=0664 --checksum=sha256:c17fa4cb658e3583423e915b9f3acc01cceaee1860e33d59ebae66adc3a2dc0d \
  https://static.crates.io/crates/ring/ring-0.17.8.crate /crate
SHELL ["/usr/bin/dash", "-c"]
RUN \
  --mount=from=rust-base,src=/lib,dst=/lib \
  --mount=from=rust-base,src=/lib64,dst=/lib64 \
  --mount=from=rust-base,src=/usr,dst=/usr \
    set -eux \
 && mkdir -p /extracted \
 && tar zxvf /crate --strip-components=1 -C /extracted

I use this to fetch and extract a .crate. Passing the .crate directly to rustc would allow me to drop everything after the ADD :)

---

WRT git I'd say it'd enlarge the scope of this issue. But I quite like the idea as well!

[weihanglo]

we could abstract this in a way for cargo to define the filesystem so we could read from git databases directly

The performance might be a concern for reading from git versus from file system.

[weihanglo]

squashfs could be another possibility.