nix copy support to exclude paths which are already in cache.nixos.org

[mlyxshi]

Is your feature request related to a problem? Please describe.

Currently, nix copy will copy whole closure. For many users who self-host cache server, it is unnecessary to upload paths that have been cached in cache.nixos.org to a self-host server. This default behavior wastes time and storage, especially when nix copy a nixosConfiguration to self-host cache server.

Describe the solution you'd like

I use traefik to reverse proxy minio server. Therefore, I make a router rule which will forward GET http://MYHOST/BUCKET/HASH.narinfo to a proxy server as a workaround. This proxy will GET http://cache.nixos.org/HASH.narinfo first. If status code is 200(already in official cache), it just pipes cache.nixos.org response to client so that nix copy will not upload this path. If status code is 404, it pipes MYHOST response to client.

Test Result

1. nix copy whole closure
   Full log: https://github.com/mlyxshi/flake/actions/runs/3798651110/jobs/6460501893
   Time: 16min 58s
   Storage: 383.5MiB

2. nix copy exclude paths which are already in cache.nixos.org
   Full log: https://github.com/mlyxshi/flake/actions/runs/3798783525/jobs/6460750038
   Time: 27s
   Storage: 23.6MiB

All in all, nix copy add a flag that supports to exclude paths already in cache.nixos.org will benefit a lot for self-hosted users. Although, I have found a workaround, It would be more convenient to have official support and the implementation is not complex

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Priorities

Add 👍 to issues you find important.

[SuperSandro2000]

Essentially you are asking to substitute paths from cache.nixos.org instead of copying them?
Because currently it is not possible by design to have missing dependencies of a store entry and would be considered inconsistent and broken.

[mlyxshi]

Essentially you are asking to substitute paths from cache.nixos.org instead of copying them

nix copy support --substitute-on-destination on SSH stores. Actually, it's not related to this feature request. More specific, the feature request is related to HTTP cache server and s3 cache server. The logic is that at the beginning of nix copy, nix will calculate which paths need to be uploaded base on the result of GET http://CACHEHOST/HASH.narinfo. I think user experience can be improved by also considering the result of GET http://cache.nixos.org/HASH.narinfo. In real-life scenario, every machine has substituter cache.nixos.org. Therefore, I think it is unnecessary to upload the path which are already in cache.nixos.org

have missing dependencies of a store entry and would be considered inconsistent and broken.

I have tested that

nix-env --store /mnt -p /mnt/nix/var/nix/profiles/system --set $closure \
--extra-trusted-public-keys "cache.mlyxshi.com:qbWevQEhY/rV6wa21Jaivh+Lw2AArTFwCB2J6ll4xOI=" \
--extra-substituters "https://cache.mlyxshi.com" 

can install and boot the entire system with no errors

[mlyxshi]

implemented https://github.com/zhaofengli/attic

[arianvp]

I'd love this to be implemented in Nix natively...