Better insulate stdenv

[jonringer]

In downstream overlays, a lot of these packages can be freely overriden, causing non-obvious stdenv rebuilds. Might be nice to create a stdenv package set which has really slimmed downed nix expressions to just achieve stdenv and avoid being affected by downstream overlays

[GrizzlT]

I discovered that this is most likely due to the following code:

  # The complete chain of package set builders, applied from top to bottom.
  # stdenvOverlays must be last as it brings package forward from the
  # previous bootstrapping phases which have already been overlayed.
  toFix = lib.foldl' (lib.flip lib.extends) (self: {}) ([
    stdenvBootstappingAndPlatforms
    stdenvAdapters
    trivialBuilders
    splice
    stdenvPackages
    otherPackageSets
    aliases
    configOverrides
  ] ++ overlays ++ [
    stdenvOverrides ]);

The actual stdenv is really only defined in stdenvOverrides and is thus subject to all user-defined overlays. This achieves maximum flexibility to change as one wishes but indeed forces stdenv to rebuild.

I'm in the process of rethinking stdenv and package set composition.

[jonringer]

I was playing with the idea of bootstrap tools being more complete, and thus requires less "nixification"/overlay magic to work.

The other desire here is to reduce how much it takes to eval stdenv. As of writing, stdenv alone takes some 320 drvs/sources to compile:

[14:59:25] jringer@jringer-5560-nixos ~/projects/nixpkgs (master)
$ nix-instantiate -A stdenv
/nix/store/asnfc1fbhcpnrhnk4kkrs5zrb5gdf5qs-stdenv-linux.drv

$ nix-store -qR /nix/store/asnfc1fbhcpnrhnk4kkrs5zrb5gdf5qs-stdenv-linux.drv | wc -l
325

By using the natures of FODs, we could lessen the coupling between stdenv and bootstrapping so that nix just needs to some stdenv polishing before being usable.

[GrizzlT]

I think the stdenv should be as small as possible. Is it even desirable to override the tools in stdenv? For the C compiler, it's quite easy to extract all the different versions in a separate repository (or a different package set (scope) in corepkgs). The coreutils and other tools that are also in stdenv might be more tricky to extract separately.

Maybe this simply requires a slightly different approach to bootstrapping. I really like the elegance of the theory on bootstrapping in multiple stages. I might be able to combine this with my new scope composition idea to only touch stdenv in practice in the programming language adapter packages.

From what I've seen so far, the bootstrap files include a C compiler, a subset of coreutils and patchelf. Is it even necessary to require anything else from the bootstrapper than these precompiled files and a working fetchurl derivation? Shouldn't a higher layer be concerned with providing the actual nixified packages?

[jonringer]

Is it even desirable to override the tools in stdenv?

I think this has a pro/con of most of these tools can be curated similar to other packages, thus they don't need to be treated different when doing updates or reviews. The main downside is that it can be surprising when you get stdenv rebuild on something like libidn2

Maybe this simply requires a slightly different approach to bootstrapping.

Yea, I would be fine with increasing the bootstrapping bundle size if it meant that it was more autonomous. The downside is that it will likely be harder to "create" stdenv's in general, so supporting many platforms will increase in difficulty.

But mainly including https://github.com/ekala-project/stdenv/blob/master/generic/common-path.nix in the bootstrap tools, then they don't need to be separately added.

Shouldn't a higher layer be concerned with providing the actual nixified packages?

In principle yes, but there's also many costs. I'm not saying one is more strictly correct. But I am saying that I would like to investigate if pushing more into the bootstrap would have compounding benefits that are beyond what I mentioned already.

[GrizzlT]

I would be fine with increasing the bootstrapping bundle size if it meant that it was more autonomous.

I did not necessarily mean to change the bootstrap bundle, I don't think it should contain more than the minimal set of tools to bootstrap a c compiler. Adding more to the bootstrap bundle might improve rebuilds or development but will most likely not matter when the bootstrap results get cached.

As of writing, stdenv alone takes some 320 drvs/sources to compile

I would like to see a decrease here as well.

The current idea I'm experimenting with (which I think would be a good basis for the ekapkgs poly-repo approach) is to distinguish different scopes where recursion is allowed inside of a scope, but not across scopes.

Choosing the scopes so that dependencies can be followed, intuitively improves both ease of maintenance and ease of contribution. For example, it is theoretically useless to put the result of a c compiler derivation in the same recursion scope as the scope that computes different python packages. Recursion for python packages is useful, but a c compiler should not depend on these python packages (recursively).
My intuition is that actual recursion between packages is limited to a significantly smaller subset of the packages in nixpkgs. This makes me think that limiting the recursion of stdenv to a single scope will improve readability.

I see a lot of benefits to a proper use of scopes that is uniform across all of ekapkgs. However, as a downside, mkDerivation will have to change. This might be an opportunity to think about improvements on mkDerivation as well.

A source of confusion and issues are override semantics for derivations. Do you have ideas for that? Do you have any intuition for how overrides should merge with the base derivation configuration?