Dynamic derivations-based lock-based fetching

[roberth]

Project description

Consider that

• Lockfiles bloat the Nixpkgs tarball #327064

It would be great to be able to fetch a lock file from anywhere instead, using a fixed-output derivation.
However, in many cases, we'd then use the Nix expression language to analyze, which is not something we're allowed to do in Nixpkgs, because it creates an undesirable data dependency on builds, known as import-from-derivation (IFD)

RFC 92 proposes a generalization of derivations that is weaker than IFD, avoiding the undesirable data dependency, ensuring that the evaluator can exit before builds commence, among other things.
Specifically what it does allow is for a derivation to compute another derivation as its output, and for expressions to reference outputs of outputs (where the latter is a computed derivation).

This new capability allows for a new fetching strategy to be used.

flowchart LR
    make-fetched-dependencies -- references --> fetched-lockfile
    fetched-dependencies --references --> fetched-lockfile["lock file (FOD)"]
    make-fetched-dependencies -- has output --> fetched-dependencies
    subgraph dynamic
      fetched-dependencies --references --> fetched-dependency-1["fetched dependency 1 (FOD)"]
      fetched-dependencies --references --> fetched-dependency-n["fetched dependency n (FOD)"]
    end
    the-actual-build --references --> fetched-dependencies
    the-actual-build --calls builtins.outputOf on --> make-fetched-dependencies
    fetched-dependency-1 -.....- fetched-dependency-n

Loading

Caveats and limitations

• This is a long-term project. Initially solutions like this will be proofs of concept, which inform the development and prioritization, and are a stepping stone towards production use in Nixpkgs.
• RFC 92 is unreleased due to a bug it had in the scheduler
  • this bug results in a store path not being scheduled sometimes, and is otherwise harmless
  • workaround: invoke Nix again
  • you can run Revert "Revert "Adapt scheduler to work with dynamic derivations nix#9415
    • use something --store ~/rfc92-store to avoid your "production" daemon which doesn't have dynamic derivation support yet
    • nix run nix/dynamic-derivations -- --store ~/rfc92-store build <argument>
• This strategy is currently most suited for use with language package managers that fetch pre-built dependencies, such as the JavaScript and JVM ecosystems. Go is probably also ok, because it's acceptable to build dependencies in the main derivation anyway.
• The dynamic part is, as you can see in the diagram, a graph. In order to create the dependencies (fetched-dependency 1..n), you'll need to enable recursive-nix.
  • Instead of recursive-nix's almost complete nix-daemon, we'd like to provide a simple protocol with only "store add" functionality, such as creating derivations and "normal" content-addressed store paths, but no "build" or "realise" commands. This is not implemented, so use recursive-nix instead, limiting yourself to only adding derivations.

Context

Get in touch with @Ericson2314 @tomberek @roberth, who are also in the nix-dev matrix room

• [RFC 0092] Computed derivations rfcs#92
• dynamic-derivations experimental feature
• Lockfiles bloat the Nixpkgs tarball #327064
• Revert "Revert "Adapt scheduler to work with dynamic derivations nix#9415

---

Add a 👍 reaction to issues you find important.