More intelligent subflake locking.

[tejing1]

Is your feature request related to a problem? Please describe.
flake.nix:

{
  inputs.child.url = "path:./child";
  outputs = {self, child}: {
    inherit child;
  };
}

child/flake.nix:

{
  outputs = {self}: {
    foo = 1;
  };
}

$ nix eval .#child.foo
1
$ # so far, so good
$ sed -ie 's/1/2/' child/flake.nix
$ nix eval .#child.foo
warning: Git tree '/mnt/persist/share/data/tejing/work/tmpflake' is dirty
1
$ # huh, ok, maybe the dirty tree support doesn't extend that far.
$ # I guess that's at least kind of understandable
$ git commit -am 'change foo' &>/dev/null
$ nix eval .#child.foo
1
$ # this is super unintuitive. it really should be 2 at this point
$ nix flake update --commit-lock-file  &>/dev/null
$ nix eval .#child.foo
2
$ # finally!

Describe the solution you'd like
Nix should recognize that the subflake is already locked by the parent flake's rev, and thus doesn't need its hash put into flake.lock at all, removing any possibility of a parent and child from different commits interacting. Ideally the dirty tree case should work too.

Describe alternatives you've considered
Nix could try to keep the hash in flake.lock up to date somehow, but it fundamentally doesn't have a hook into the proper time to do that, since git commit doesn't call nix.
The parent flake could invoke the child by builtins.getFlake, rather than as an input, but currently all methods of doing that suffer from similar problems, and you wouldn't have access to input overriding functionality.

[thufschmitt]

That’s indeed really annoying. I don’t think using the parent’s lock would be a good idea (if only because it’s too much of an overapproximation), but maybe we could be able to specify that some inputs should be systematically refreshed when running a nix command. So you’d have

{
  inputs.child.url = "path:./child";
  inputs.child.autoUpdate = true;
}

and any nix command would first call the equivalent of nix flake lock --update-input child.

[Kha]

Some prior discussion (though tracking this in an open issue is probably a good idea): #3978 (comment)
I really don't think path-relative, flake-local inputs should be locked at all, as they are reproducible by virtue of being included in the flake source, and there is no way to even fetch a stale locked version.

[aameen-tulip]

Some prior discussion (though tracking this in an open issue is probably a good idea): #3978 (comment) I really don't think path-relative, flake-local inputs should be locked at all, as they are reproducible by virtue of being included in the flake source, and there is no way to even fetch a stale locked version.

This makes sense except for the case of nix build github:owner/repo?dir=directly-to-a-subflake#bar in which case we actually do want the lock for the sub-flake.

[tejing1]

This makes sense except for the case of nix build github:owner/repo?dir=directly-to-a-subflake#bar in which case we actually do want the lock for the sub-flake.

It feels like you're misunderstanding here. There are 2 meanings of 'lock' at play. In all cases, the subflake's lock file should be used. The question is whether the hash of the subflake gets recorded in the lockfile of the consuming flake, when the consuming flake is actually stored in the same git commit. In the case you brought up, there is no consuming flake, and thus the kind of locking in question isn't even happening.

[aameen-tulip]

Gotcha. Well as someone who's been bitten badly by subflake locking I definitely agree with "make relative subflakes work more better".

Relocking every directory in my tree before committing is obnoxious, and honestly I only remember to do it post-hoc when I get a crash later on another box.

The project that I fully can't get them to work with uses "merge and squash" PR as the only option, so relative paths will always blow up unless there's something I misunderstood about how to manage the locks.

From what I can tell the changes to the commit hash made by the squash cause a hash mismatch that isn't recognized as "dirty", but also fails to match the locked hash. If you touch something to dirty the tree it will "do what I want", but that's obviously not workable.

Honestly what I want is "experimental commands in every directory", with a repo global lockfile.

[blaggacao]

Inspired, I submitted this work around.

[roberth]

#6530 improves subflake locking, but is currently blocked on evaluation regressions. We've made some progress merging some stable parts of that PR, so perhaps its solution for subflakes could be extracted and merged independently too?

• Fix/improve subflake handling.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flake-lockfile-update-loop-when-having-dependent-flakes-in-a-monorepo/27937/7

[mikepurvis]

Are subflakes as described in the original post workable at all under today's Nix? I tried to adopt that exact structure and it all worked great locally, but then on Hydra building from gitlab URL it hit evaluation errors (with Nix 2.13.3):

in job ‘thingy.x86_64-linux’:
error: cannot fetch input 'path:./thing?lastModified=1&narHash=sha256-tk7BEyl8BW4OSrT%2fRCAhYBzJx3LQJxPcV1e2MFRl7kM=' because it uses a relative path

This seems to be a complete showstopper as far as being able to use multiple flakes in the same repo for anything other than local development.

EDIT: Looks like the master issue for this is #3978.

[tomberek]

the best fix workaround for this at this moment seems to be to use https://github.com/divnix/call-flake or something similar using call-flake.nix:

{
   inputs.call-flake.url = "github:divnix/call-flake";
   outputs = inputs: (inputs.call-flake ../..).something.something {};
}

I'm aware of this issue. Though I'm not clear on the best path to fix it. @roberth has had proposed some lockfile changes that can help fix.

[roberth]

the best fix for this at this moment

This is not a fix, but rather a workaround. For instance, it can't work with --override-input.

I think @edolstra has written an experimental fix, but he hasn't opened a viable PR for it yet.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-team-member-suggests-removing-flakes-data-suggest-it-isnt-a-good-idea-please-remove-the-experimental-flag-instead/54959/117

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/my-nix-subflake-does-not-resolve-nixpkgs/55941/4

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/my-nix-subflake-does-not-resolve-nixpkgs/55941/8

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/my-nix-subflake-does-not-resolve-nixpkgs/55941/1

[rasmus-kirk]

What's the status on this?

[roberth]

I really don't think path-relative, flake-local inputs should be locked at all, as they are reproducible by virtue of being included in the flake source

Nix 2.26 includes

• Improve support for relative path inputs #10089

I don't see anything in this thread that isn't covered by that (despite plenty of room for scope creep 😄)
Some questions about whether the parent lock should be involved, but I think that would be a separate issue if the current solution isn't adequate. The thread in #10089 does mention desyncing of transitive inputs in the parent lock, as those transitive dependencies are still copied, but we could consider that to be a special case of #7730.

So I think this one can be closed.

[mikepurvis]

My use case is met with the changes made in #10089. I had a generated flake that included a small metadata component that I wanted to be able to later use overrides against in order to do hash comparisons.

So like, generated timestamp flake X and generated timestamp flake Y (from say, an hour later) would trivially compare as unique if the timestamp itself is part of the content, but by putting it in a subflake, I can override one to the other when comparing, and that tells me if the content itself has changed (and therefore if the new timestamp flake should be committed and shipped).