--eval-store and faster closure copying #5048
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In particular, this now works: $ nix path-info --eval-store auto --store https://cache.nixos.org nixpkgs#hello Previously this would fail as it would try to upload the hello .drv to cache.nixos.org. Now the .drv is instantiated in the local store, and then we check for the existence of the outputs in cache.nixos.org.
With this, we don't have to copy the entire .drv closure to the destination store ahead of time (or at all). Instead, buildPaths() reads .drv files from the eval store and copies inputSrcs to the destination store if it needs to build a derivation. Issue NixOS#5025.
This adds a new store operation 'addMultipleToStore' that reads a
number of NARs and ValidPathInfos from a Source, allowing any number
of store paths to be copied in a single call. This is much faster on
high-latency links when copying a lot of small files, like .drv
closures.
For example, on a connection with an 50 ms delay:
Before:
$ nix copy --to 'unix:///tmp/proxy-socket?root=/tmp/dest-chroot' \
/nix/store/90jjw94xiyg5drj70whm9yll6xjj0ca9-hello-2.10.drv \
--derivation --no-check-sigs
real 0m57.868s
user 0m0.103s
sys 0m0.056s
After:
real 0m0.690s
user 0m0.017s
sys 0m0.011s
|
How can I enable |
|
You can't since it's not a config option and I'm not sure if it makes sense to turn it into one, since it could have unexpected interactions with scripts that use |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/tweag-nix-dev-update-16/14379/1 |
Ericson2314
reviewed
Sep 30, 2021
Comment on lines
+109
to
+112
| static ValidPathInfo read(Source & source, const Store & store, unsigned int format); | ||
| static ValidPathInfo read(Source & source, const Store & store, unsigned int format, StorePath && path); | ||
|
|
||
| void write(Sink & sink, const Store & store, unsigned int format, bool includePath = true) const; |
There was a problem hiding this comment.
This shouldn't be part of the core data type, but instead defined like and with the other serializers in src/libstore/worker-protocol.hh.
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/tweag-nix-dev-update-32/19865/1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the Nix daemon is running remotely and is accessed via a high-latency connection, an operation like
nix build nixpkgs#hellocan be very slow because the client first needs to send a lot of .drv files one-by-one to the remote, each requiring a round trip. To make this faster, this PR does the following:It adds an option
--eval-storethat allows the evaluation to be done in a local Nix store. This is also useful for store that can't build, like binary caches. For instance, you can now do:It adds a new worker protocol interface to add multiple store paths in the same call. On a link with a 50 ms delay, this speeds up the copying of the
hello.drv closure from 57.8s to 0.7s. Note that this disables parallel copying (the use ofprocessGraph()with a thread pool incopyPaths()is commented out).Fixes #5026.
Partially fixes #5025.