detachable builds

[nrdxp]

Is your feature request related to a problem? Please describe.
It would be nice in the new remote build case nix build --store ssh-ng://$url if we could optionally detach the client from the remote builder and still allow the build to continue on the remote, at least after the derivations have been uploaded.

We could then proceed with additional work on the client side and perhaps ping the remote for build status if we need the artifact later on.

Describe the solution you'd like
At a high level it should be as simple as a boolean flag: --detach-from-daemon

Describe alternatives you've considered
I know there is talk of completely redesigning the worker protocol, and obviously it would be fantastic if we could have something more efficient with it's own high-level http API, but that's a lot of work and potentially a long way off, where this solution could fill the gap in the mean time.

Additional context
nixbuild.net has it's own implementation of this via its keeps-builds-running setting and it's quite nice, but we can't make use of it generically in std-action because users should rightfully be able to configure any remote builder they want to.

This would be useful in our case where we have an initial job that performs all the Nix evaluation up front and then sends work via derivations to individual build matrices. We could spawn the matrices and allow them perform any setup tasks while waiting on the builder to finish with any artifacts it requires without blocking on all builds, since some will take longer than others.

[blaggacao]

The problem is the red arrow.

[Ericson2314]

A more asynchronous interface would also be useful decoupling Hydra from ssh://. I am curious what the Bazel remote execution protocol does for this; that is something @thufschmitt has thought about at least somewhat.

[thufschmitt]

I am curious what the Bazel remote execution protocol does for this

Not sure, but I suspect once a build is started it doesn't care about the client any more and will keep running no matter what. Which wouldn't be a great default for Nix, but could indeed be interesting as an option

[rickynils]

Not sure, but I suspect once a build is started it doesn't care about the client any more and will keep running no matter what. Which wouldn't be a great default for Nix, but could indeed be interesting as an option

This is basically what nixbuild.net does, if keep-builds-running has been activated. For the Nix client, this works fine. It wouldn't be too hard to implement that in the current Nix protocol. However, for the end user, it will be tricky to track these builds, right? nixbuild.net has an API that can be used to query builds, but there is nothing like that for plain Nix. I guess you could perhaps try running nix log xyz.drv on the remote builder and see if you get anything?

A solution could be that you get some build id back from the remote builder, that can be used later on to query the build status. I guess you would also need to get the address of the remote builder back, to know which host to query. I think a solution along those lines would work without any significant changes to current protocol, just a new op for querying build status.

I'm also interested in finding out how this has been solved in other places. In general, when you have any kind of long-running remote operations it feels so wasteful to let your CI/CD runners just hang around waiting.

[thufschmitt]

Yes. The reason I mentioned it as “not a great default” is that it's probably not what you want in the general case. If my local Nix client triggers a build on my local Nix daemon, then I'm generally happy to just have to close the client to stop the build.

But for a number of use-cases (hydra included I guess) that would be a great feature to have. I could imagine a scenario like:

$ nix build some-package --detach --store ssh-ng://foo.bar
Build id 51365 started. Re-attach with `nix job attach --store ssh-ng://foo.bar 51365` or stop it with `nix job stop --store ssh-ng://foo.bar 51365` 
$ nix job attach --store ssh-ng://foo.bar 51365
... showing the build log and waiting for the build to finish ...
^C
$ nix job stop --store ssh-ng://foo.bar 51365
Job stopped

[nrdxp]

Not a bad flow, I was also thinking you could execute other commands, so that if you are at the point in your work where all the setup is done and you have nothing to do now but wait, you can just do something like this:

$ nix copy --from ssh-ng://foo.bar /nix/store/*-foo.drv
Build of `foo` is currently running with id 3453, will continue when finished

[Ericson2314]

See also #3813 for giving identifiers to different attempts.

Even if the vanilla daemon nix shies away from supporting this stuff, it could make sense to standardize an expanded protocol for all the CI implementations which do already need to track attempts, do things async, etc.

[edolstra]

A while ago I made a design for a new protocol that supports this. I.e. clients connect, submit a request to build a derivation, get a token back, and then can asynchronously poll the build status/completion. Builds would get cancelled if the client doesn't check the status within N seconds.

[7c6f434c]

Builds would get cancelled if the client doesn't check the status within N seconds.

Hopefully _a_ client (in the sense that all the clients waiting on the same build get equal rights unlike the current asymmetric approach that sometimes leads to annoyance)