Move build loop into the client

[edolstra]

Currently, when using the Nix daemon, the whole dependency graph machinery (i.e. Worker, DerivationGoal etc.) is done in the daemon. However, this doesn't work well with --eval-store(#5025) since the daemon doesn't have access to the eval store used by the client. It also doesn't allow fixed-output derivations to be run in the context of the client (where it has access to stuff like SSH keys forfetchGit`).

So we should move the build logic into the client. Worker already has been generalized to work on any Store, so the main remaining change is to have DerivationGoal and SubstitutionGoal call the daemon to do the actual build/substitution.

[Ericson2314]

I hope this be done in a way to improve remote builders more generally? e.g. one can already do nix-build --store local --builders 'auto - - 1 1'

Per #5025 (comment) it sounds like you had similar thoughts. I hope we can do just that!

[Ericson2314]

Also, I hope this doesn't need to be mandatory? E.g. if I am using a big build farm, I do want the build farms to do the scheduling.

I think being able to open "reverse" connections would help such that the remote store can use the client as a build remote for those fixed-output derivations / remote store for anything the client has already built.

[rickynils]

@edolstra Can you clarify why "since the daemon doesn't have access to the eval store used by the client" is a problem? After evaluation, the client will copy all .drv files to the daemon, right?

Like @Ericson2314 I hope this would be an optional thing. For https://nixbuild.net, it is beneficial to just receive a big pile of .drv files that we can schedule to our own liking.

The idea about being able to open a reverse connection to the client, and use the client as an additional substituter sounds like a great way of solving the problem of fetching fixed outputs that needs SSH keys etc.

[Ericson2314]

@rickynils Incidentally, I think the "reverse connection" use-case is the best concrete reason to switch to an off-the-shelf RPC protocol. You really want to multiplex streams to avoid bad liveness issues of traffic for requests in one way blocking the other, and stuff gets complex quick.

[rickynils]

@Ericson2314 Yeah, I agree. Any ideas on how we can get started? :)

[Ericson2314]

As I said on twitter to you :), first we must test more!

#5753 can be merged. #5754 needs to also be done, because #5753 needs to be backported and my backport ran into sneaky variable issues.

Perhaps you want to try seeing what's going wrong with #5754?