Recursive Nix support #3205
Recursive Nix support #3205
Conversation
|
Assuming that the inner build is quite large. How would the build be distributed to the same set of remote builders as the outer build? One thing I was wondering is, if it would make sense for the inner build to just return a drv file in $out instead, and then let the outer scheduler update its build plan accordingly. This would be a bit closer to IFD but where the evaluation happens in a builder instead of all in the client. |
|
@edolstra I am worry about exposing In short, while I think
I don't want to be in a position where people write a bunch of stuff that uses |
|
I've made an initial version of a nix-ccache flake: https://github.com/edolstra/nix-ccache. It provides a wrapper around gcc/g++ that executes the compilation of the preprocessed source in a recursive nix-build call. |
This allows Nix builders to call Nix to build derivations, with some
limitations.
Example:
let nixpkgs = fetchTarball channel:nixos-18.03; in
with import <nixpkgs> {};
runCommand "foo"
{
buildInputs = [ nix jq ];
NIX_PATH = "nixpkgs=${nixpkgs}";
}
''
hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })')
$hello/bin/hello
mkdir -p $out/bin
ln -s $hello/bin/hello $out/bin/hello
nix path-info -r --json $hello | jq .
''
This derivation makes a recursive Nix call to build GNU Hello and
symlinks it from its $out, i.e.
# ll ./result/bin/
lrwxrwxrwx 1 root root 63 Jan 1 1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello
# nix-store -qR ./result
/nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
/nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5
/nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo
This is implemented as follows:
* Before running the outer builder, Nix creates a Unix domain socket
'.nix-socket' in the builder's temporary directory and sets
$NIX_REMOTE to point to it. It starts a thread to process
connections to this socket. (Thus you don't need to have nix-daemon
running.)
* The daemon thread uses a wrapper store (RestrictedStore) to keep
track of paths added through recursive Nix calls, to implement some
restrictions (see below), and to do some censorship (e.g. for
purity, queryPathInfo() won't return impure information such as
signatures and timestamps).
* After the build finishes, the output paths are scanned for
references to the paths added through recursive Nix calls (in
addition to the inputs closure). Thus, in the example above, $out
has a reference to $hello.
The main restriction on recursive Nix calls is that they cannot do
arbitrary substitutions. For example, doing
nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10
is forbidden unless /nix/store/kmwd... is in the inputs closure or
previously built by a recursive Nix call. This is to prevent
irreproducible derivations that have hidden dependencies on
substituters or the current store contents. Building a derivation is
fine, however, and Nix will use substitutes if available. In other
words, the builder has to present proof that it knows how to build a
desired store path from scratch by constructing a derivation graph for
that path.
Probably we should also disallow instantiating/building fixed-output
derivations (specifically, those that access the network, but
currently we have no way to mark fixed-output derivations that don't
access the network). Otherwise sandboxed derivations can bypass
sandbox restrictions and access the network.
When sandboxing is enabled, we make paths appear in the sandbox of the
builder by entering the mount namespace of the builder and
bind-mounting each path. This is tricky because we do a pivot_root()
in the builder to change the root directory of its mount namespace,
and thus the host /nix/store is not visible in the mount namespace of
the builder. To get around this, just before doing pivot_root(), we
branch a second mount namespace that shares its /nix/store mountpoint
with the parent.
Recursive Nix currently doesn't work on macOS in sandboxed mode
(because we can't change the sandbox policy of a running build) and in
non-root mode (because setns() barfs).
Derivations that want to use recursion should now set requiredSystemFeatures = [ "recursive-nix" ]; to make the daemon socket appear. Also, Nix should be configured with "experimental-features = recursive-nix".
edolstra
force-pushed
the
recursive-nix
branch
from
eb7131b to
69326f3
Compare
Very cool! This looks a lot like what @layus talks about at NixCon. How costly is it to run every compilation in nix-build, though? Perhaps we need some heuristic to determine whether a C file is big enough to be cacheable, otherwise we impose a constant builder setup for every .c file. |
|
A quick unscientific measurement suggests the overhead is ~0.15s per GCC call on my laptop. (This also depends on the size of the preprocessor output, since it needs to be copied to the Nix store.) This is enough to make configure scripts much slower, so right now there is a special check to disable building through recursive Nix when the input is called "conftest". A heuristic like you suggest might be better. |
|
@volth you just reinvented import from derivation :) But a big benefit of recursive (to me at least) is trying to leverage eval less not more, i.e. nix-exprs can just be one unprivileged way to get drv files. Other than that< I think you might prefer NixOS/rfcs#40.
Ret-cont recursive also does this.
Never need to go stdout->store path either. |
Well, |
|
Hi, I am searching for a working example with recursive nix. Does it work with the nix in nixpkgs-unstable? Are there docs already? |
|
@kolloch have a look at https://github.com/NixOS/nix/pull/3205/files#diff-e9794c2e4d63a50bf65a1a0ce0873a19 for an example. The feature is hidden behind a feature flag. In terms of Nix releases, it looks like it's only available in master at the moment. |
|
I assume that does that mean my nix daemon also has to be from master? |
|
Yes, in this case the daemon needs to be updated as well since it controls the build sandbox. |
|
When will this land in stable? |
|
When Nix 3.0 will be released. Or use |
Fixed-output derivations (FODs) would be part of a recursive Nix solution for fetchNodeModules. I'd like for it to leverage Nix's ability to cache downloads in the form of FOD and share the downloaded modules in the form of store paths. Instead of prohibiting FODs, we could stop the console logs from going to the recursive Nix socket and send them directly to the calling derivation's log instead. This way, the sandboxed build can only determine whether an FOD is fetchable. That's still sufficient to extract information from the network bit by bit, so we'd also have to "prohibit" failures by killing the parent derivation whenever a recursive derivation fails. It's worth noting that the ret-cont solution does not have this problem, but can support the |
|
any chance for backporting to 2.3 ? |
|
The example does not build for me. I wonder why? I'm running NixOS from nixpkgs master with and I have copied the example derivation from the PR into Can anyone see which requirement I'm failing to fulfil? I'd love to use recursive Nix. |
|
@lukego I also have the same issue. Have you been able to find a solution? |
|
@jkarni Yes, I have been able to get past this problem and onto the next problem with recursive nix, that it hangs in large builds 😁. See #7297 for more on that and a (hopefully) working small example for reference. I am ahem 80% sure that the problem above was resolved by adding |
I can confirm that the system feature was needed for me as well as enabling the experimental feature! Why is that the case though? Where did you find documentation about this? |
|
When using remote builders, you wouldn't want to schedule a derivation that requires the experimental feature onto a machine where it's not enabled, so Nix must require both. |
|
Ah, so
I understand that, but would have expected it to be on the user to ensure the experimental nix feature was enabled (either using system features, or simply by ensuring all their remote builders have the experimental nix feature enabled). |
This allows Nix builders to call Nix to build derivations, with some limitations.
Example:
This derivation makes a recursive Nix call to build GNU Hello and symlinks it from its
$out, i.e.This is implemented as follows:
Before running the outer builder, Nix creates a Unix domain socket
.nix-socketin the builder's temporary directory and sets$NIX_REMOTEto point to it. It starts a thread to process connections to this socket. (Thus you don't need to havenix-daemonrunning.)The daemon thread uses a wrapper store (
RestrictedStore) to keep track of paths added through recursive Nix calls, to implement some restrictions (see below), and to do some censorship (e.g. forpurity,
queryPathInfo()won't return impure information such as signatures and timestamps).After the build finishes, the output paths are scanned for references to the paths added through recursive Nix calls (in addition to the inputs closure). Thus, in the example above, $out has a reference to
$hello.The main restriction on recursive Nix calls is that they cannot do arbitrary substitutions. For example, doing
is forbidden unless
/nix/store/kmwd...is in the inputs closure or previously built by a recursive Nix call. This is to prevent irreproducible derivations that have hidden dependencies on substituters or the current store contents. Building a derivation is fine, however, and Nix will use substitutes if available. In other words, the builder has to present proof that it knows how to build a desired store path from scratch by constructing a derivation graph for that path.Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network, but currently we have no way to mark fixed-output derivations that don't
access the network). Otherwise sandboxed derivations can bypass sandbox restrictions and access the network.
When sandboxing is enabled, we make paths appear in the sandbox of the builder by entering the mount namespace of the builder and bind-mounting each path. This is tricky because we do a
pivot_root()in the builder to change the root directory of its mount namespace, and thus the host/nix/storeis not visible in the mount namespace of the builder. To get around this, just before doingpivot_root(), we branch a second mount namespace that shares its/nix/storemountpointwith the parent.
Recursive Nix currently doesn't work on macOS in sandboxed mode (because we can't change the sandbox policy of a running build) and on Linux in non-root mode (because
setns()barfs).This PR also adds some ccache-like functionality to Nix's makefiles that wraps GCC calls in Nix derivations to enable caching and remote builds. This requires recursive Nix when you want to do this inside a Nix build.
Implements #13.