Make sources.nix configurable

[nmattia]

Until now I've refused to add configuration to sources.nix because I really like:

let sources = import ./nix/sources.nix; in ...

and because in most cases it shouldn't have to be configurable. Having to pass in an empty attrset {} in most cases is really not nice. However after discussions with @zimbatm and @NinjaTrappeur I've realized we could have e.g. this:

let sources = (import ./nix/sources.nix).__configure { useBuiltinFetchers = false; }; in ...

Here most users will never have to care about the configuration, but if you need to tweak how packages are fetched, you can feel free to go for it. I have a POC here: https://github.com/nmattia/niv/blob/b69df92ce13f3b9f9d5344b24ecde6ae945c41d6/nix/sources.nix

I'm thinking about having the following configuration fields:

• useBuiltinFetchers: specifies whether builtins.fetchurl et a. should be used, defaulting to true. Hydra users can set it to false. Would solve Fallback on builtin fetchers whenever possible #46 @nlewo and we could get rid of builtin-tarball @michaelpj . This also solves some opaque infinite recursion errors due to the heuristics we use for figuring out <nixpkgs>.
• sourcesJson: the path to the sources.json which would (partially) solve Different folder for niv init #102
• fetchers: a function name: spec: fetcher: ... which fetches the spec (the attrset coming from sources.json), or just returns fetcher spec if no special fetcher is needed. This would solve Fetch via git:// does not work #130 @tfc and fetchGit and submodules #58 @roberth (at least for the case without submodules).

Thoughts?

[michaelpj]

Having to pass in an empty attrset {} in most cases is really not nice.

I don't see this as a big deal. People do this all the time when importing nixpkgs or doing callPackage. I remember actually doing a double-take the first time I saw an import without an attrset argument afterwards!

On the other hand, I have no idea what this magic __configure thing is. Somehow it affects how the thing is imported?

My vote goes for boring functions and arguments :)

useBuiltinFetchers

I don't think this should be a global toggle. My experience is that you want to use the builtin fetchers quite deliberately for specific dependencies, if at all. So it works much better as just "another fetcher" and would be a pain if anything as a global toggle.

fetchers

I like the idea of just letting people define their own fetchers. My first thought is composability: what happens when e.g. I want to share a special fetcher with people in my company? Then I need to fetch that source and import it before I can use it. This is a very similar problem to the problem I have with nixpkgs.

I wonder whether there would be some utility in having some "bootstrap" sources that get fetched first and are available to the main fetchers?

[michaelpj]

I had an idea. What if we allowed the user to provide overlays that could (mutually) define sources and fetchers?

Bear with me, there's a few pieces.

Suppose I define the following sources.json:

{
    "my-company-stuff" : { ..., type = "builtin-tarball" },
    "other-source" : { ..., type = "tarball" }
}

Let's say that this gives rise to the following sources-overlay.nix (we probably wouldn't actually generate another file, but I want to give it a name):

self: super: {
    my-company-stuff = builtins.fetchTarball ...
    other-source = self.tarball-fetcher ...
}

other-source uses tarball-fetcher which hasn't been defined yet, but the idea is that it will come from another overlay.

Now, niv itself provides base-overlay.nix, which defines

• a nixpkgs source, one way or the other (e.g. using the current heuristics)
• the fetchers that are defined in terms of nixpkgs, e.g. the tarball fetcher.

self: super: {
    nixpkgs = ... # heuristics for picking a nixpkgs from the specs if none is provided
    tarball-fetcher = (import self.nixpkgs {}).fetchTarball;
}

Now, I can finally provide an overlay my-overlay.nix that rebinds nixpkgs to come from my-company-stuff:

self: super: {
    nixpkgs = (import super.my-company-stuff).nixpkgs
}

Now, if we evaluate the overlays [ base-overlay.nix sources-overlay.nix my-overlay.nix ] then what should happen is that the nixpkgs reference in base-overlay.nix gets overridden to be the nixpkgs defined in sources-overlay.nix, and thus we end up using the tarball-fetcher from that nixpkgs to fetch other-source. And we shouldn't get infinite recursion.

---

I haven't tried this, but I think it would work. Downsides that I can see:

• Kind of complicated, probably need to steal a couple of functions from lib/fixed-points.nix.
• Fetchers are in the same attrset as sources. I think you can make this work with nested attrsets, but I'd have to try it.

If people are interested I could try whipping up a prototype.

[michaelpj]

I made a POC in #137, which IMO works quite nicely.

[nmattia]

Thanks so much @michaelpj ! I had a quick look but that wasn’t enough to fully wrap my head around the idea, I’ll give more feedback tomorrow!

[michaelpj]

It might not make much sense 😅 I can also push my examples, tbh that's probably a good idea.

[zimbatm]

Not following closely but I have a few observations I want to share:

Most of the time, I only want to use the builtin fetcher to bootstrap nixpkgs. The rest can be using build fetchers since they are faster to use (in my experience). The only other reason to use a builtin fetcher would be if it requires some access credentials that are available in my user environment.

I want to avoid evaluating nixpkgs twice because it's slow and takes a lot of memory.

I usually want to be able to initialize nixpkgs with my own custom config and overlays. So I want a way to get the nixpkgs version from niv, initialize it, and then pass it back to niv to use with the build fetchers.

One possible implementation of this could be to pass the nixpkgs initializers to niv and let niv to the bootstrapping internally:

let pkgs = import ./nix/sources.nix (sources: {
  config = {};
  overlays = [ sources.home-manager ];
});
in pkgs

Or externally:

let
  sourcesBuiltins = import ./nix/sources.nix {};
  pkgs = import sourcesBuiltins.nixpkgs {
    config = {};
    overlays = [
      (_: super: { sources = import ./nix/sources.nix { pkgs = super; }; })
      sourcesBuiltins.home-manager
    ];
  };
in
  pkgs

Something like that.

[nmattia]

Ok, had some time to dig more into it. I like it! I'd like to retain the interface of #134 (comment) though, so we could pass the overlays as

let sources = (import ./nix/sources.nix).__configure { overlays = [ ... ]; }; in ...

or maybe even have two:

let 
  sources = import ./nix/sources.nix;
  sourcesSimple = sources.__configure { useBuiltinFetchers = true; };
  sourcesMoreCustomizable = sources.__overlays [ overlay1 overlay2 ];
in ...

since the former can be implemented in terms of the latter. That way we can use overlays when necessary (though is it really necessary?), we don't break backwards compatibility and finally we can still use the simple import.

[michaelpj]

I prefer the explicit argument passing to the magic attribute, but I'd live. If we're going to have one, let's just have one. I think a simple interface and an advanced interface are fine, but multiple advanced interfaces is too much.

That way we can use overlays when necessary (though is it really necessary?)

Apart from the use-cases i gave, I'm pretty sure the overlays approach is powerful enough to let people do whatever they want. So that avoids us coming back to this issue when someone has some weird new thing they want to do.

Having some shortcut options for common things also seems fine, although I'm not sure what we'd have. I still think the global builtin fetcher toggle is questionable. However, we can add those on top of the overlays easily enough if some patterns become very common!

[roberth]

You could define sources.__functor to keep the invocation simple (e.g. sources {}) and still be backwards compatible. Basically any new function interface you come up with can be layered on top of the current API this way.

This is a good thing, because it means you can design the new sources.nix interface from scratch, as long as it's a function!

You can even have a backward compatible deprecation of the attrset interface this way.

[nmattia]

ok, I really like this overlay + functor solution. Gonna take a stab.

[leotaku]

Maybe this could be combined with the efforts currently going on in #132? I think it would be quite good UX to have sources.nix be something like this:

let
  callSources = import 
    (builtins.fetchUrl "https://github.com/nmattia/niv/blob/MY_VERSION/nix/api.nix");
in

callSources ./sources.json {
  someConfigureArgs = "someValue";
  overlays = self: super: {
     foo-fetcher = ...;
  };
}

callSources in this case returns a special attrs that can be customized further using e.g. __configure or __functor. I'm just spitballing here, but I hope my suggestion is relevant to the discussion.

[michaelpj]

Did you decide not to go with the overlay solution in the end?

[nmattia]

@michaelpj that's still the idea! Given that the overlay approach is pretty powerful and low-level, I didn't want to make it the only option. It'll look something like this:

let sources = import ./nix/sources.nix { sourcesFile = ...; overlay = ... ; } ; in ...