Optimize closure copying for high-latency remotes

[edolstra]

Currently copying closures with lots of small files (like .drv closures or NixOS system closures) is slow because every store path is copied separately and requires a round-trip to the remote. Sending the entire set of missing paths in a single call (like with the old nix-store --export) is much faster.

[Ericson2314]

I would also like to try out QUIC some day so we can send files 1-1 in parallel streams without head of line blocking too.

In any event, we might want to allow stores that don't care about references being intact to avoid doing the transfer in (partial) order.

[vcunat]

QUIC helps with cheaper handshakes and on packet losses (the head-of-line problem), but I expect neither is really significant in practice (for nix remotes), and we suffer mainly from those deep dependency graphs and the way we work with them.

[gytis-ivaskevicius]

Maybe we could have some sort of batching implemented? I am thinking something along the lines of gateway over s3 bucket which would add required batching operations (I don't think s3 on its own is flexible enough)

Also, may I add that additional functionality like permission control as part of this gateway could be added in the future. (I think feature as such would be neat)

[kvtb]

The issue with copying tons of small store paths over a high-latency link is long pause before copying.
I do not know what is going on, likely checking for which paths are already present on the target.