nix shell: set MANPATH for installables that have a man dir

[sternenseemann]

The major change introduced in this commit is that nix shell will extend
MANPATH with the /share/man directory of all installables present in the
shell. This means that the support for man pages in nix shell becomes
more portable. Currently man pages are discovered by an undocumented
(at least not in its man page) feature of man-db's man(1) which also
checks PATH to heuristically find prefixes to search for man pages.
This is not supported by mandoc's man(1), but MANPATH is supported by
both.

This allows us to partially revert the changes made in
c87f4b9 as well: Since we don't need to
rely on man-db's PATH behavior for discovering man pages, we can only
add a bin directory to PATH if it exists.

Tested with both mandoc and man-db:

• nix shell -f '<nixpkgs>' mblaze -c man mshow
• nix shell -f '<nixpkgs>' libunwind.devman -c man libunwind

[edolstra]

The problem with this is that there are numerous such environment variables (e.g. XDG_DATA_DIRS, PYTHONPATH, PKG_CONFIG_PATH) and it wouldn't be very elegant to add ad hoc support for each of them to nix shell.

[grahamc]

Maybe this is something mkShell or stdenv.mkDerivation should do. 

On April 15, 2021, GitHub ***@***.***> wrote: The problem with this is that there are numerous such environment variables (e.g. XDG_DATA_DIRS, PYTHONPATH, PKG_CONFIG_PATH) and it wouldn't be very elegant to add ad hoc support for each of them to nix shell.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4702 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe- auth/AAASXLGCRY7KY5CQBTOIJITTI3JEDANCNFSM42VHFLVA>.

[sternenseemann]

The problem with this is that there are numerous such environment variables (e.g. XDG_DATA_DIRS, PYTHONPATH, PKG_CONFIG_PATH) and it wouldn't be very elegant to add ad hoc support for each of them to nix shell.

From what I understand nix shell is intended for temporary shells where a certain program is available, not for development environment. This is why we don't reuse the derivation environment anymore, right? We need PATH for execution and to me MANPATH is in scope since it is required for user documentation. Also we already try to support this by adding directories to PATH which don't exist, but this relies on man-db specific behavior.

I think setting up MANPATH and possibly XDG_DATA_DIRS or whatever GNU info uses to discover pages is different from setting up things for development environments like PYTHONPATH.

Maybe this is something mkShell or stdenv.mkDerivation should do.

That would work for the old nix-shell, yes, but all nix shell has is ad hoc support for certain environment variables, with variables really meaning just PATH.

[edolstra]

The problem is that we haven't really decided what the semantics or use case of nix shell is supposed to be. Some more discussion here: #4609

I think setting up MANPATH and possibly XDG_DATA_DIRS or whatever GNU info uses to discover pages is different from setting up things for development environments like PYTHONPATH.

This is already a pretty open-ended set of environment variables, especially if you consider packages that provide plugins. E.g. should nix shell update QT_PLUGIN_PATH?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/improving-flakes/12831/47

[stale]

I marked this as stale due to inactivity. → More info

[SuperSandro2000]

bump

[SuperSandro2000]

@thufschmitt can you help to bring this forward?

[Ericson2314]

I enqueued it on project board.

[fricklerhandwerk]

Triaged in Nix maintainers meeting:

• There are a lot of variables we would have to bless independently, this doesn't scale.
• @thufschmitt: if someone wants to write a consistent proposal, we'll gladly review it
• @Ericson2314: this opens up many architectural questions, especially how we can delegate authority away from Nix to provide solutions for particular use cases where Nix is too generic
  • declarative single-origin profiles are an entirely different problem from ad hoc combining packages, they are different use cases
    • none of them should be a concern for core Nix
  • propose closing this
  • doesn't make sense to keep the problem within our domain of responsibility and at the same time delegating it away
    • @fricklerhandwerk: agree, we've been dodging nix shell subjects lately but still changes would be bottlenecked on maintainers
• have to discuss this further

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-06-30-nix-team-meeting-minutes-67/29835/1

[sternenseemann]

If you are taking that stance, I think nix shell is already to far—how is MANPATH different from PATH?

I am actually very open to explore this hardline view, as I believe it would hold the key to solve one problem we have, namely the dubious status of (development) environments. We currently inherit development environments from derivations' build environments which is arguably wrong. The symptoms of this are the variety of withPackages facilities that are only intended for interactive use and not (usually) utilised by derivations (to be built) and the fact that mkShell yields a derivation that can never be built.

nix run and nix shell are steps away from this that are to be applauded, but ultimately they are not flexible enough (or rather not flexible at all) which is why it is very difficult to use them as basis for a development-environment-first alternative to nix develop. nix develop is a misnomer, as it is a derivation debugging facility, strictly speaking.

I feel like the solution may be to have a second type of special Nix value in addition to derivation that does not declare a build environment and instructions, but an environment modification (how to encode that is another question, you could probably look to direnv, shadowenv or invent something). That would mean that user (nixpkgs) code can prescribe what variables are set and Nix could ignore this topic completely. (An alternative to a new value type would be to support special nix-support files, e.g. a shadowlisp script that is called to do the env setup.)

In any case, if we are serious about “delegating authority away from Nix”, a lot of things need to be re-evaluated. By the same logic, it feels questionable that the following things are Nix commands:

• nix develop—is okay of course in the sense of debug-derivation, but is development environments really Nix's authority?
• nix profile
• nix shell (nix run seems okay in comparison)
• nix bundle
• nix fmt

[sternenseemann]

To be even more provocative: Shouldn't shells, any kind of environments just be nix run .#shell or $(nix-build -A shell --no-out-link)/bin/shell where it is totally opaque to Nix how it works?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-cli-working-group/30517/1

[thufschmitt]

Discussed during the Nix team meeting:

@sternenseemann since you're already climbing down that rabbit hole, would you be willing to write down in more detail / discuss the idea of splitting away nix develop from derivations you expose in #4702 (comment) ? That would be a much better long-term solution than adding environment variables in an ad-hoc fashion

Discussion log

• Suggestion: Since Sterni already started sketching out a more principled proposal, we could try to sneak him into expanding it into something more concrete
  • @tomberek: That’s expanding the scope of the discussion quite a lot, shouldn’t we first focus on the small-scope problem at hand?
  • @regnat: I’d rather know where we’re going before expanding it in any way. Possibly we can allow MANPATH once we know we have an escape route

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-07-24-nix-team-meeting-minutes-74/31116/1

[SuperSandro2000]

I tried rebasing this PR on nix 2.19 and tested it successful with the example commands from above.

diff --git a/src/nix/run.cc b/src/nix/run.cc
index ea0a17897..5491f4a7b 100644
--- a/src/nix/run.cc
+++ b/src/nix/run.cc
@@ -111,16 +111,22 @@ struct CmdShell : InstallablesCommand, MixEnvironment
         setEnviron();

         auto unixPath = tokenizeString<Strings>(getEnv("PATH").value_or(""), ":");
+        auto manPath = tokenizeString<Strings>(getEnv("MANPATH").value_or(""), ":");

         while (!todo.empty()) {
             auto path = todo.front();
             todo.pop();
             if (!done.insert(path).second) continue;

-            if (true)
-                unixPath.push_front(store->printStorePath(path) + "/bin");
+            auto pathString = store->printStorePath(path);

-            auto propPath = CanonPath(store->printStorePath(path)) + "nix-support" + "propagated-user-env-packages";
+            if (accessor->maybeLstat(CanonPath(pathString) + "bin"))
+                unixPath.push_front(pathString + "/bin");
+
+            if (accessor->maybeLstat(CanonPath(pathString) + "share" + "man"))
+                manPath.push_front(pathString + "/share/man");
+
+            auto propPath = CanonPath(pathString) + "nix-support" + "propagated-user-env-packages";
             if (auto st = accessor->maybeLstat(propPath); st && st->type == SourceAccessor::tRegular) {
                 for (auto & p : tokenizeString<Paths>(accessor->readFile(propPath)))
                     todo.push(store->parseStorePath(p));
@@ -128,6 +134,7 @@ struct CmdShell : InstallablesCommand, MixEnvironment
         }

         setenv("PATH", concatStringsSep(":", unixPath).c_str(), 1);
+        setenv("MANPATH", concatStringsSep(":", manPath).c_str(), 1);

         Strings args;
         for (auto & arg : command) args.push_back(arg);

[fricklerhandwerk]

a second type of special Nix value in addition to derivation that does not declare a build environment and instructions, but an environment modification

A few spontaneous ideas:

• Encode it as an attribute set and add functionality to the command line that translates it to something the current shell can eat
  • Advantages:
    • Declaration can be agnostic to the shell
    • Will support additional shells once Nix supports it
    • Easy to compose multiple packages in one environment: just append/override the environment variables in sequence
  • Disadvantages:
    • Requires Nix to know a lot of things about this
    • Imposes policy from below in the stack (I think this is a disadvantage, because it binds Nix to every policy decision forever)
    • This will likely break down in unfixable ways once there is some subtle divergence
    • Can't possibly support arbitrary shell features such as adding functions to the environment
• Let package authors produce environments for every shell they care about and run them like any other executable through nix run
  • In the simplest form, everything is manual: nix run foo#shells.fish which can (but doesn't have to) be aliased to nix shell foo#fish
  • Optionally add convenience to detect the current shell
  • Advantages:
    • Very simple and transparent on the Nix side
    • Moves domain knowledge up the stack where it belongs
      • Up the stack you can provide any formalism or abstraction you need to make it shine
    • Allows for really cool things out of the box, such as non-trivial default configurations
      • Those could even be overridden on the fly if we keep the nix-shell -p semantics where the argument is a literal Nix expression.
      • Then you could do crazy stuff like nix run nginx#'shells.fish.override { rootDir = "/home/user/test"; }'
  • Disadvantages:
    • No obvious fallback if a shell isn't specified
      • Except the regular build output put into $PATH, which is what's being discussed here
      • Maybe it's not even a problem when stated explicitly: If you want it, add it to your package.
    • Requires package authors to think about yet another thing in their package
      • Can be alleviated with support tooling in Nixpkgs
    • Not obvious how to compose environments

Bonus points for separation of concerns if the all of the command line business that's not nix run <source> <attr> doesn't even have to be part of Nix. Nixpkgs knows best what a package is and how it's structured, it could have its own command line following its own conventions, however package maintainers please.

[thufschmitt]

@fricklerhandwerk your second proposal seems to be more or less #7501, which everyone seems to broadly agree with. So 💯 to it.

[fricklerhandwerk]

My ramblings #4702 (comment) were missing the fact that only nix run and nix develop take exactly one thing to run an executable. We recently discussed how to refer to attributes from different expression sources, and nix shell or whatever you want to call it naturally needs to support multiple such references.

It's not clear how one would compose an environment from entire shell runs in a simple and obvious way. In any case it would need some sort of contract to implement against, and I'd prefer it to be very small, both in terms of API surface as well as amount of custom code needed to implement it.

• One weird and probably stupid idea would be always adding Nix to the shell's environment, and expecting any given shell environment to take parameters in order to run nix run <tail of environments>.
• Another, possibly more sensible option, would be sticking to POSIX shell (i.e. requiring environments to be POSIX shell scripts), and leave room for wrappers to translate that to other shells. Then environment composition would mean running their scripts in sequence. The problem arises what will run them (does Nix ship with its own sh?), and who's responsible for making the result available to arbitrary shells.
• The sanest I currently can come up with is, for nix shell to indeed just expect an attrset of environment variables in the package attrset and concatenate those literally. But then again, why does Nix have to specify what that should look like? It's exactly such an arbitrary, inflexible forever-policy that IMO we should avoid in Nix, and instead maintain such encodings in Nixpkgs. A practical issue with having this in Nix is that it would ship with a convenient but strictly less powerful means of composition (which by itself is not wrong, but) that works very differently from what you'd do in Nix expressions, the primay way of doing real things.

In short, I suggest to leave buildEnv to Nixpkgs, and reduce the Nix command line interface to essentially nix run, nix build, nix eval, and nix store (roughly in descending order of abstraction), explicitly encouraging the community to build and organise useful tooling around that.

[Aleksanaa]

@fricklerhandwerk Should we draft an RFC to see what can be developed from this idea and what's the attitude of the community?

[iFreilicht]

In short, I suggest to leave buildEnv to Nixpkgs, and reduce the Nix command line interface to essentially nix run, nix build, nix eval, and nix store (roughly in descending order of abstraction), explicitly encouraging the community to build and organise useful tooling around that.

I agree with the general architectural idea, but I would hate to see some of these very useful commands go. I use nix shell and direnv's integration with nix daily, they're the main reasons why I use Nix in the first place. I would be fine with these being maintained "elsewhere", but I'm wondering how to bring them to users conveniently.

The main challenge I see is to provide an easy way to add functionality to one's installation. There must be some sort of canonical default entrypoint to start from. I concur that Nix shouldn't

[...] ship with a convenient but strictly less powerful means of composition [...] that works very differently from what you'd do in Nix expressions, the primay way of doing real things.

so this default should be declarative and work exactly like composition works everywhere else in Nix, so no nix profile, no additional scripts or other shenanigans.

Maybe the solution can simply be to source Nix differently. In addition to adding

. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh

to /etc/bashrc or /etc/zshenv, we could consider automatically creating a default user flake (or regular nix file) in .config/nix/, (similar to how configuration.nix is created by default in NixOS), and sourcing the resulting environment of that something like

source "$(nix build --no-link --print-out-paths path:$HOME/.config/nix/user#userEnv)"
# OR
source "$(nix build --no-link --print-out-paths --file $HOME/.config/nix/userEnv.nix)"

after or during the shell startup by default.

This would replace nix profile for user-wide installations of programs in a 100% declarative way, it would be completely replaceable (or extensible) by other tools like home-manager, and how the environment is actually built would be the responsibility of nixpkgs' buildEnv.

If we find a good mechanism for this entrypoint, I believe this architectural change would solve all the issues that motivate it without negatively impacting the user experience at all.

(We still might need nix profile, though just as a barebones tool for managing generations and rollbacks. Installing or upgrading anything would be handled by additional tools that could be part of userEnv by default)

[fricklerhandwerk]

(@iFreilicht) I'm wondering how to bring [some of these very useful commands] to users conveniently.

My general idea on this is to essentially port all the porcelain into Nixpkgs and, from there, provide installers for various targets that embed Nix. Then you'd install a Nixpkgs release rather than a Nix release which implicitly sets you up with some Nixpkgs. That could also solve the compatibility struggles between Nix and Nixpkgs to some degree. But I haven't put much thought into how exactly all of that would work. (Another potentially beneficial side effect of doing it that way would be that we could then formally eject the installation business from the Nix code base and let distributors take care of that, according to their needs. Then all we'd have to do is provide documentation on the moving parts that need to be considered. My impression is that the Nixpkgs community would be much more capable of carrying that load than Nix maintainers are at the moment.)

Edit: @Aleksanaa I don't think this needs an RFC, but a prototype to get an idea of how it feels to work with it and how much effort it is to maintain and evolve.

[fricklerhandwerk]

I gave the whole thing another series of thoughts, and studied the direnv source code to see what we can learn from it. Here's another proposal, narrowing down #4702 (comment) and addressing considerations from #4702 (comment):

• Add rules in Nixpkgs:
  • derivations can expose a flat attribute set env with environment variables and their values, following some restrictions:
    • no function values
    • only flat lists
  • derivations can expose shellHook.<shell> scripts in the respective language
• Add Nixpkgs library functions to
  • compose lists of env attribute sets
    • plain values are overwritten in order
    • lists are concatenated as configured per attribute name
      • e.g. : by default, and null could mean "overwrite in order"
  • convert attribute sets to shell-specific instructions to set up that environment
  • create shell-specific scripts from a list of derivations based on env and shellHook.<shell> that will
    • stack the environments
    • convert the result to instructions for setting the new values
      - there probably needs to be a flag specifying whether to leave the original value for list attributes, as in PATH=value:$PATH
    • make instructions to preserve values of changed environment variables
    • make instructions to revert the environment to the original state
    • concatenate the hooks
  • wrap the shell-specific scripts in convenience executables
    • observing a --pure flag that will discard surrounding the environment

That way we could produce arbitrary shells from Nixpkgs packages. One could also reproduce build environments based on NixOS/nixpkgs#324789. Maybe with some trickery we could build a wrapper that takes some information from a failed build and starts the shell in the temporary build directory.

Most of the complexity of direnv could be skipped entirely. A Nixpkgs-specific variant would merely have to manage allowed directories and handling asynchronous rebuilds. A project-specific configuration file such as .nixenv would only have to specify which environment source (Nix file and attribute) to use for the given directory.

What do you think?