Redesign of the nix command line

[Profpatsch]

Initial writeup by @edolstra: https://gist.github.com/edolstra/efcadfd240c2d8906348

[Profpatsch]

I did a little writeup with my thoughts & proposals:
https://gist.github.com/Profpatsch/d5c8e1ccd68baab0e3f0

In that I address the problem of search path clashes @edolstra mentioned in the General section.

[Ericson2314]

(Commenting here as @copumpkin requested)

---

I strongly agree with @Profpatsch's points. To be fair there, there is a use for imperative package management, as articulated by @lethalman in #684. Since under @Profpatsch's plan the profile is always defined declarative, and since @lethalman's usecases boil down to pinning the exact version of a package, I think we should speak of "pinned" vs "regular" packages.

---

Start a development shell based on ./default.nix or ./shell.nix:

$ nix shell

(Hm, do we want a special case like this?)

I'm reminded of git changing the default push settings. I suggest we disable this by default, and then have a setting to enable this power users can enable. In the name of orthogonality, I also propose that the setting be that the namespace is queried unless default.nix/shell.nix is present, and have it apply to at least nix build and nix use in addition to nix shell. [I don't think nix install should have this, because that command is more global, and it violates our new attribute path agenda].

[Ericson2314]

More broadly, I see a convergence in the way userland nix and nixos are configured and used. We've already talked about using the nixos module system both for user-mode services (e.g. on OSX), and for configuring packages. Likewise there was recently a thread to the mailing list about decoratively pinning nixpkgs for nixos in https://www.mail-archive.com/[email protected]/msg17827.html .

Ultimately It would be great if root and normal users alike can define and enable service and packages in the same module system, and the same CLI tools can assist with querying and editing the config files.

Converging all this will take a great deal of work, but I think a few decisions can be made now to help the process go faster. In the thread I linked, @edolstra mentioned we could use fetchTarball, or move the module system in nix itself (presumably into the corepkgs). We have a similar choice for the UI itself---to use the git terminology, we could have just the minimal plumbing commands in nix, and porcelain commands like nix install kept elsewhere for more rapid iteration. [This also provides a way to make nix repl work without bundling the repl with nix].

[zimbatm]

Is it possible to cut down the proposal to the essentials ? It's good to show the end-goal but for implementation it might be a bit daunting. How would we go on implementing all of this ? Are we going to port one command at the time and update the docs accordingly ?

One thing that might feel inherent but that needs to be specified is that thenix executable wouldn't be doing much, it would behaves like git and just dispatch sub-commands to nix-<command> executables in it's libexec/ and PATH.

[vcunat]

First my strongest points:

• Very nice ideas overall.
• There are too many similar top-level commands, as noted already. I believe it's better to group them more, e.g. create more hierarchy or introduce options switching some behavior changes.
• The commands should have short-cuts, even though the long form should be preferred in documentation. Some people like to have short aliases for frequent commands. One way is adding aliases like nix -i bash, nix -q foo, another way is allowing to use a unique prefix of a command like btrfs fi def for btrfs filesystem defragment. Enhanced alternative: allow defining aliases, e.g. like git.
• Commands that take attribute names should accept arbitrary nix expressions instead. This could be extended even to package updates, e.g. if I nix install 'foo.override { myOption = bar; }', I want to keep the override even after nix upgrade foo.
• Installation actions should have an interactive mode, perhaps even by default if connected to terminal: first show essentially --dry-run and then ask for confirmation whether to proceed.
• Implementation: I assume that to simplify Get rid of the Perl dependency #341 we'll only do the new C++ implementation for the new command scheme, and the old commands would require perl and we would remove them anyway after several months of overlap.

Now some replies:

• @itkovian: nix build is what adds stuff into the store (and a symlink into the current directory). To most people "installation" is expect to do much more than that, e.g. make $PATH contain the executables, etc.
• @madjar: I've got a set of packages that I use daily, and I want those consistent and often updated. Then I have some I rarely use and sometimes huge, so updating them often would be rather wasteful, e.g. LibreOffice.
• @CMCDragonkai: that sounds like ctags and similar tools. Perhaps such functionality belongs into the REPL, but note that the "location" is often difficult to pinpoint, e.g. the binding is done in all-packages.nix but most of the code is in another file.
• @Profpatsch: the nixpkgs.pkgs. prefix seems redundant. Anyone can introduce extra namespaces via packageOverrides already. (I use it to define personalized easy to update custom package sets.) There's only a single nixpkgs=... in $NIX_PATH and that should be the default namespace, I believe.

More detailed thoughts of mine, ordered according to the original proposal:

• nix search --name foo is rather confusing terminology, as we'll likely want the attribute name to be the (unqualified) name
• nix install
  • --declarative: rather --auto-update or similar. Calling it "declarative" just seems strange to me (from a fresh point of view).
  • -f: perhaps replace by -I to handle all $NIX_PATH changes more consistently?
  • machine:package : also support the other direction?
• nix rebuild: wouldn't that better be somewhere in nix upgrade? For example, nix upgrade by itself working only on auto-upgraded packages and with -a or --all doing the "usual" upgrade?
• nix upgrade: maybe I like update better. From some mainstream distro I've fixed upgrade as something big and dangerous... but we could e.g. make them synonymous.
• nix list and nix status: these two seem to work too similar to warrant extra (top-level) subcommands. Perhaps call it nix query or nix info instead. (Also noted by @trishume.)
• nix uninstall: maybe unlink? (analogy to UNIX* traditions around file deletion)
• nix rollback: could be also undo, but rollback seems a more popular term in general. Improvement: generations might remember which one came before, as currently nix-env --rollback; nix-env -i foo; nix-env --rollback does an unintuitive thing.
• nix use, nix sandbox, nix shell: very similar, again. Maybe nix use --sandbox or --pure or something...
• nix query-options: perhaps this doesn't warrant another subcommand. I'd join this with the above, e.g. nix query --options bash and nix query --plugins firefox.
  • Note: perhaps among this the various variants should be handled. I'd think that instead of firefox-esr attribute we should have an option, e.g. variant="esr". Similarly for the barFull packages, etc.
  • Note: for options, especially global ones, there's a question whether to propagate them to dependencies. We might consider bringing nixos and nixpkgs closer to each other.
• nix build: again the -f considerations.
• nix repl and nix eval: similar, maybe nix eval -i or --interactive, though "repl" might be too well remembered already.
• nix make-store-derivation: why not nix-build --drv instead? It should do the same thing, only stop after instantiation.
• nix fetch-url: maybe have nix fetch instead, accepting multiple schemes: https?, ftp, git, ...
  • add --unpack as fetchzip equivalent.
• nix fetch-closure, nix send-closure: maybe not needed when we allow nix build --from machine /some/store/path and --to similarly.
• nix query-closure, nix closure-size: why not nix query --closure...?
  • TBD answer: it should be able find out closure from binary caches, even without fetching the paths.
• nix path-info, nix substitutes: why not under nix query?
• nix source: perhaps add --rollback to address some concerns.

[Profpatsch]

The commands should have short-cuts

+1 for unique substrings of command names, -1 for alias definiton (that’s what the shell is for, and the git folks got it all wrong!)

Commands that take attribute names should accept arbitrary nix expressions instead

What should be in scope by default?

Installation actions should have an interactive mode

I find that absolutely crucial with a “destructive” package manager like e.g. pacman, but what use is it with nix?

Anyone can introduce extra namespaces via packageOverrides already

Yet, that is not useful if one uses two versions of e.g. nixpkgs, e.g. 15.09 and master (which is a pretty standard usecase).

[glaebhoerl]

@vcunat I notice that wherever you suggested a third level in the hierarchy, you did it as an --option; but couldn't it be turtles all the way down, so that sub-subcommands look and work the same as subcommands, e.g. nix query closure instead of nix query --closure?

[vcunat]

-1 for alias definiton (that’s what the shell is for, and the git folks got it all wrong!)

@Profpatsch: how do you simply define aliases for sub-commands in bash?

What should be in scope by default?

I imagine it like this: the default $NIX_PATH points to some channel so I can use nix install bash directly, and if I want to use a different nixpkgs tree (channel), I'll have other ways to do that, among them e.g. modifying $NIX_PATH by -I options.

I find that absolutely crucial with a “destructive” package manager like e.g. pacman, but what use is it with nix?

Personally, I sometimes like to see what would happen beforehand, instead of doing it and the rolling back immediately.

Anyone can introduce extra namespaces via packageOverrides already

Yet, that is not useful if one uses two versions of e.g. nixpkgs, e.g. 15.09 and master (which is a pretty standard usecase).

Even today you should be able to do this:

{
    packageOverrides = pkgs: {
        pkgs-1509 = import /nix/var/nix/profiles/per-user/root/channels/nixos-15.09 {};
    };
}

Other ways are e.g. adding nixpkgs-15.09=... into $NIX_PATH instead and importing that. Of course, we could automatize something like this when adding channels, in some way...

(And even today you can relatively simply get rid of the prefixes; the only question is what should be the default. There was a similar discussion on the mailing-list a couple days ago.)

---

@glaebhoerl: for clarity reasons, I assume it's better to prefix by -- in cases where it's optional to specify a sub-command. For example, if we allowed nix query git bash to display information about multiple packages, it would seems strange to allow nix query closure bash and instead I feel it would be better to have nix query --closure bash.

[CMCDragonkai]

@vcunat Yea I think the functionality should be in REPL or in the manual. See for example the Haskell documentation, and how if you look up the source for any particular function, it points to the actual implementation. For the REPL if it could show the type signature (even though nix is untyped, it could just be manually specified, like a help page for that exact command), that would be great.

[cstrahan]

Once some of the initial discussion settles down, I would suggest creating a new repo to host a design document. PRs would then be issued to get feedback on proposals before updating the doc.

I'm really enjoying the level of discourse here, but I think it's important that we have an authoritative document so we're all on the same page, and have a more directed approach to converging on a final, actionable design.

Ideally, we'd have @edolstra and a couple others that we could trust to review and give feedback on each proposal, merging in the changes if they sound good.

A deadline for the design might also be a good idea, to avoid this work languishing.

Thoughts?

[cstrahan]

Alternatively, we could just have @edolstra comb through all the commentary, and in true BDFL fashion, develop the final design to his liking :). Whatever works best for us / @edolstra.

[vcunat]

Yes, a very good point. Best agree on basics of the UI before implementation (perhaps except for some prototypes).

[Profpatsch]

how do you simply define aliases for sub-commands in bash?

alias myalias='nix subcommand'

It’s nice to see the difference between built-in and not. Otherwise you sit at a different system and wonder why the most basic commands are missing …

Even today you should be able to do this:

packageOverrides is dependent on nixpkgs, though. I don’t like the idea that most functionality of nix only works in conjunction with the default package set. In the same manner I don’t think we should hardcode the string nixpkgs anywhere in nix.

[CMCDragonkai]

Based on my email:

Just wanted to ask, is this package search enhanced with showing package
parameters planned to be added to the standard command line Nix query
tools? Or if it's already available and I haven't found it.

I would like to suggest that any querying functionality should include the ability to show package parameters.

[CMCDragonkai]

Another suggestion, an inline way of overriding package parameters instead of using packageOverrides inside ~/.nixpkgs/config.nix. Something like nix-env -i firefox -o '{ blah = true; }'. Where the attribute set could be a recursive attribute set evaluated inside a function that takes the old pkgs, like pkgs: rec { blah = true; }.

Oh this already has been suggested. NVM.

[Profpatsch]

Something like nix-env -i firefox -o '{ blah = true; }'.

That is of course only an instance of the more general case of -E 'with import <nixpkgs> {}; firefox.override { blah = true; };'. We will have to do very close evaluation what special cases we want to introduce, since we are going to have to support them for all eternity.

I’m in strong favor of not depending on anything nixpkgs specific in the new tool.

[Ericson2314]

Derivations and Nix sees them do not reflect any configuration options, and I'd like to keep it that way. I earlier proposed that this be developed in nixpkgs. But really there are 3 layers: nix itself, modules system tools and config idioms (other parts of nixpkgs/lib perhaps), and the actual nixpkgs packages. I agree nothing in the Nix repo should depend on nixpkgs at runtime, but whether we move this and module system into nix, or put tools in nixpkgs, we are combining those 3 layers into 2.

[EmmanuelOga]

Disclaimer: just an implementation detail.

You may want to consider docopt. There's a cpp port and some examples of how you may implement a huge multi-command app like git.

Greets

[bjornfor]

Don't forget to look at guix for inspiration. I've read some of its CLI documentation, it looks very good. Annoyingly good actually... can I haz?

Here is one (out of many) really useful looking guix sub-commands:
http://www.gnu.org/software/guix/manual/guix.html#Invoking-guix-refresh

[vcunat]

The code now contains nix command with some basic functionality (in src/nix/). I've been using it to see what it feels like. I was missing some flags, e.g. -kKj, so I looked into adding them, but I see they were moved into LegacyArgs which suggests they're not planned to be supported. @edolstra: can you confirm/explain that? I would assume at least some of those would be kept. Another point is that nix build doesn't print results or create those symlinks, but that seems just like not implemented yet.

BTW, working with the code is relatively hard for me as I can see just the code, almost without any explanation of intention etc., which makes it a little problematic to understand, especially as it's WIP. (When trying to understand header definitions, I often had to look into implementation and call sites to see what they're for.)

[CMCDragonkai]

If we're planning a multi command app like nix build and nix shell... etc. There should be an easy to to create nix aliases like how git allows git aliases.

[vcunat]

That was already discussed above in more detail.

[matthiasbeyer]

When this will be implemented, will it be done in a backwards compatible manner? So that there are scripts around in my PATH which are nix-env -i and map to nix install for example, including printing a big fat "WARNING" or something that the nix UI has changed?

---

Another idea: Some of you might know my nixos-scripts which are basically scripts around nix-{env, shell, build} to provide some more functionality (like diffing of generations). These things will be covered in the new UI as far as I can see, but maybe you guys can get some more inspiration from my scripts on what to include (for example nix container new --template ~/container-templates/apache-container.template.nix or something like that). Just suggesting!

---

Edit: Another thing I really like to say that the idea for aliases is really great, but we should take this a small step further: the nix binary should be able to find nix-<foo> binaries in the $PATH and provide them as subcommands nix <foo>. This way, one could for example write a command nix ui to start a ncurses interface or nix gui to start a graphical interface for the nix command or even more tools I cannot think of yet.

[domenkozar]

nix-env will coexist with new nix command. They both use the same api while exposing a different layer to the user.

[copumpkin]

I thought nix-env's name-centric approach was mostly getting removed in the new UI?

[Ericson2314]

@copumpkin I hope @domenkozar means the same C++ API internally, and the new commands simply won't use the name-centric parts.

---

Is there a nixos-setting for using nixUnstable so we can try this out more easily?

[vcunat]

I don't see why not add a switch for name-based resolution.

[vyp]

I haven't read all the previous discussion so I'm not sure if this has already been brought up but I had a quick look and it didn't seem like it. With nix-shell we can use it as a shebang interpreter (this is outlined in the man page for it):

#! /usr/bin/env nix-shell
#! nix-shell -i bash -p bash
echo hello world

$ ./hello
hello world
$ 

But with nix shell it doesn't work:

#! /usr/bin/env nix shell
#! nix shell -i bash -p bash
echo hello world

$ ./hello
/usr/bin/env: ‘nix shell’: No such file or directory
$

This is because the POSIX script invocation mechanism only allows one argument to appear on the #! line after the path to the executable, and so it reads nix shell as one "nix shell" string (I have nixUnstable installed). So I was wondering if this was already considered somewhere?

Guile uses a 'meta switch' as a workaround for this. So if this hasn't been considered maybe something like that could also be implemented for nix, to make it continue to work as a shebang interpreter?

Edit: I also found https://github.com/shlevy/long-shebang.

[Anton-Latukha]

Maybe it can be just:

nix-shell
---
#!/bin/sh
nix shell

So manually you can use nix shell seamlessly as all other commands.
And nix-shell for scripting.

[bobvanderlinden]

Having a distinction between the two seems very reasonable. It might even be called nix-script in that case? That way there at least is no confusion between the two (with dash, without dash).

[rnhmjoj]

@bobvanderlinden It exists already, nix-script

[vyp]

@Anton-Latukha But that's sort of my point though, we can't use nix-shell if it becomes deprecated.

[matthiasbeyer]

I'm not sure whether this is still active here, but I'd like to propose a hook interface.

The idea behind that is a use-case (of course): I run custom scripts when rebuilding my system. The relevant ones are the channel-update and switch which both have a single purpose: Build the update (channel or system) and tag my configuration with a special git tag (special as in it identifies the generation by number) so I can see in my configuration repository which generation was build from what git commit.

Having a hook executed for each action in the rebuild process would simplify that a lot.

I think about these hook points:

• Starting to build a new generation
• Rebuiling a generation failed
• Rebuiling a generation succeeded

And parameters should give the script the ability to see what gets rebuild, who did the rebuild, how it gets build (switch, boot, build, test, ...) and possibly more I do not think of right now.

[edolstra]

Don't know about those use cases, but it would be good to have a garbage collector hook to allow NixOS to remove boot menu entries for GC'ed versions.

[twhitehead]

@vcunat @Profpatsch @edolstra Really like this new interface! Quick comment on the subject of commands that take attributes being able to work with expression though.

Given that I can tell our users (Canadian HPC center) to do this to get an environment with python3

nix run nixpkgs.python3

it would be very nice if I could tell them to do this to get an environment with python3 and numpy

nix run 'nixpkgs.python3.withPackages (pkgs: [pkgs.numpy])'

instead of having to drop this sort of thing on them

nix run '(let nixpkgs = import <nixpkgs> { }; in nixpkgs.python3.withPackages (pkgs: [pkgs.numpy]))'

[Wizek]

@twhitehead, you can already simplify that to the following, can't you?

nix run '(import <nixpkgs> {}).python3.withPackages (p: [p.numpy])'

[twhitehead]

@Wizek Yes, that is more compact and avoids introducing let ...; in ..., which is nice.

I would still really like to provide colleagues and users with a command that looks like just a small extension of the basic one and avoids exposing any more of the nix language than absolutely required.

I know it may seem silly, but being able to do the basics (e.g., install/use python with a select set of packages) with as uncryptic command as possible is really important for adoption in our organization.

[twhitehead]

On the subject of being able to do the basics like installing python with a set of packages with as nonthreatening looking command as possible (i.e., gives the impression that I understand what is going on without having to learn a whole bunch), I've been wondering if even this could be improved upon

nix run 'nixpkgs.python3.withPackages (pkgs: [pkgs.numpy pkgs.matplotlib])'

For example (not suggesting this is the best way to do this, but more throwing it out as an example to get the conversation started), as withPackages (pkgs: [ ... ]) seems to have become a fairly standard construct, what if there was some syntatic sugar like so

expr1 | expr2 ... --> expr1 (dict: with dict; [ expr2 ... ])

to enable to following sort of commands

nix run 'nixpkgs.python3.withPackages | numpy matplotlib'

[hedning]

This could be done by having an option to pass regular arguments like --arg does for attribute arguments. So something like this (not sure what the best option name is though):

nix run nixpkgs.python.withPackages --arg1 'pkgs: [pkgs.numpy pkgs.matplotlib]'

[twhitehead]

Or, if the attribute selection is to remain special syntax, then it could just be extended to also be able to take an optional space separated list on the end that is translated like so

attr --> attr
attr val1 ...  --> attr (dict: with dict; [val1 ...])

allowing the fabulously clean and intuitive looking

nix run 'nixpkgs.python3.withPackages numpy matplotlib'

[CMCDragonkai]

Regardless of how the final syntax is, it would be great if all the language-specific packages had the same style of invocation.

[twhitehead]

@hedning @CMCDragonkai with regard to options and the same style of invocation, maybe it would be possible to add command line options that do common invocation things (thus also encourage further packages to implement those invocation styles).

For example, a --with-packages option could be added

nix run nixpkgs.python3 --with-packages 'numpy matplotlib'

that, given the above, would check if nixpkgs.python3 has a withPackages attribute and then invoke it like so

nixpkgs.python3.withPackages (p: with p;  [ numpy matplotlib ])

Likewise, an --override option could be added

nix use nixpkgs.python3 --override x11Support=true'

that would check for an override attribute and then invoke it like so if it exists

nixpkgs.python3.override { x11Support = true; }

This has the advantage of being fairly future proof as it provides a layer of abstraction between the user and the continuously evolving nixpkgs interfaces. Options can be added, deprecated, removed, or even just have their internal implementation details changed to continue providing the same functionality against new nixpkgs interfaces.

One pain I could see is that as nixpkgs evolves, nix would start having to check versions to provide the appropriate glue code. This suggests that possible the majority of the functionality should live in nixpkgs itself. For example, nixpkgs could provide a top-level attribute set that nix can use to extend the options it takes. Combined with a standard way of serializing option data, nix could then accept these options and invoke the corresponding nixpkgs expressions and let them manipulate the underlying nix expression that is ultimately used.

That is, nix would start with an expression for the specified attribute. This would be passed to the option function corresponding to the first option along with the serialized option data. This would give a new expression, which would then be passed to the option function corresponding to the second option along with its serialized option data and so on until there are no more options. Then nix uses the final expression to obtain the required derivation instead of the specified initial attribute.

[twhitehead]

I was thinking some more about this, and had another suggestion along these lines. What if individual packages could be passed command line options, so something like this

nix run nixpkgs.python3 --with numpy matplotlib --foo --bar -- nixpkgs.gcc

would be translated into something like this

nixpkgs.python3.cmdline { with = [ "numpy" "matplotlib" ]; foo = []; bar = []; }
nixpkgs.gcc

That is, if a set of options are specified after an attribute selection, they are collected into an attribute set and passed to a cmdline handling attribute of the selection attribute (or whatever name would be appropriate) if it exists, otherwise an x does not support command line options error is generated.

Thanks! -Tyson

[twhitehead]

There is an interesting trade off between providing a generic (more power, the ultimate being an arbitrary function transformation) and a specific interface (better error messages and such) for these things. It seems you want to be a specific as possible but no more specific. This allowing everything that needs to be done to be done while still giving the best possible error messages and such.

Most of my suggestions so far have been for generic machinery. Here is one for specific machinery instead. Maybe packages could optionally provide a options (or meta.options) attribute as with NixOS modules. This could specify what options (flags) could be provided, their types (i.e., boolean, string, etc.), and document them. All of which would make for a very good user experience (think help and error messages).

This might ultimately also dovetail nicely with also having a better config system. That is, one where options are declared in a modular manner, documented, and the user's config is verified against them in order to provide a top-level pkgs config attribute, rather than the current case of the top-level config attribute just be sucked in from ~/.config/nixpkgs/config.nix. Deeper operations (i.e., more fundamental modifications to nixpkgs) would then be left to be done as overlays.

In terms of the command line usage, you could imagine the NixOs options/config machinery merging and verifying nixpkgs.config from /etc/nixos/config.nix, ~/.config/nixpkgs/config.nix, and finally a config specification generated from the command line options to provide the top level config attribute to nixpkgs. A high-level package like python could then provide a packages list-of-strings option declaration and access it via the top-level config.python.packages attribute.

nix run nixpkgs.python --packages numpy scipy -- nixpkgs.gcc

[peti]

Can we close this ticket? It feels to me like it's not terribly useful any more. If anyone has issues with the new UI, then it's probably better to open a new, specific ticket about that.

[CMCDragonkai]

Did everything mentioned here get addressed? If not, maybe they can be packaged into a feature wish-list wiki?

[lrworth]

Since nix is a bit confusing for beginners and this GitHub issue is a main source of information, here are the release notes where this tool was introduced, and one of the few places this tool is documented: https://nixos.org/nix/manual/#ssec-relnotes-2.0

It appears that nix install doesn't exist yet (and I am not personally convinced imperative package management with nix is a good idea). Is there a plan to replace nix-env eventually? Is there a GitHub issue?

[aaronjanse]

@lrworth There's now the experimental nix profile install to do that. See https://nixos.wiki/wiki/Nix_command/profile

(yes, it's been a year, but I'm still seeing this issue in search results, so I thought I'd answer for future visits)

[bobvanderlinden]

The problem is that the new commands are not stable yet. Using them requires use of unstable nix in addition to adding experimental flags in nix.conf. newcomers will be confused. This all just adds just additional complexity to using nix which is totally unnecessary.

Personally I think this issue is still open as the new command redesign is not released yet.

[nrdxp]

Plus 1 for reopening, but there is now also a guideline for more information at least:
https://github.com/NixOS/nix/blob/master/doc/manual/src/contributing/cli-guideline.md