apptainer, singularity: precede system-level bin paths in defaultPath and fix singularity image running

[ShamrockLee]

Description of changes

apptainer and singularity now prioritize PATH from the system over those constructed from dependent packages, when it comes to the substitution of defaultPath values (the PATH hard-coded inside Apptainer/Singularity library for them to find third-party utilities). It is now constructed by the following sources, ordered by their precedence:

• systemBinPaths, a new argument introduced to specify the /**/bin paths from the system, especially those with their SUID bit set.
• The original defaultPath value, making it work out of the box in FHS systems.
• defaultPathInputs, a list of packages to form the fall-back PATH.

This change is required to enable Sylabs SingularityCE (singularity) to run images, as it requires a fusermount3 commant with SUID bit set.

The arguments newuidmapPath and newgidmapPath is deprecated in favour of systemBinPaths. Their support will be removed in future releases.

New option programs.singularity.systemBinPaths is introduced to specify the systemBinPaths argument of the overridden package. It includes "/run/wrappers/bin" even if specified empty.

The option programs.singularity.enableFakeroot is deprecated and has no effect. --fakeroot support is now always enabled as long as programs.singularity.systemBinPaths is not forcefully overridden.

This PR depends on #306656 and #306716, and currently contains their commits. The size of this PR will reduce once the dependent PRs are merged.

Together with #306716, this PR fixes #295809 and prevents the broken singularity from going into the stable release. If this fails to merge before the branch-off, I would like to have the changes backported without the deprecation warnings.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[Mic92]

There is no branch of yet. So if you get your pull request finished up soon, just merging it, will also make it available in the upcoming release. If not, we can still backport it, if the current approach is unusable without this change anyway.

[ShamrockLee]

I rebased this PR on top of master, and both apptainer and singularity seems working well.

I don't know how to test those GPG-related functionality. @SomeoneSerge @FynnFreyer, could you help test if the GPU works with these changes applied?

BTW, should we rename pkgs.{apptainer,singularity}-overriden-nixos to pkgs.{apptainer,singularity}-overridden-nixos in this PR? The renaming should not break any workflow we maintain, and we have explicitly stated that the two variant packages are for binary caching only.

[ShamrockLee]

@SomeoneSerge, could you help look at this PR? This will fix the issues regarding third-party utility access for Apptainer and SingularityCE.

[SomeoneSerge]

I am terribly, terribly sorry about losing touch on this one. I remember I had tried building the gpu tests just via the flake reference and I couldn't because the cuda image blew up in size. I'll give it another try

[SomeoneSerge]

Ok, with #323056 in I can finally run the test without modifying the diskSize arguments:

❯ (nix build --impure .#apptainer.gpuChecks.image-saxpy --print-out-paths ; echo saxpy) | xargs apptainer exec --nv
trace: warning: cudaPackages.autoAddDriverRunpath is deprecated, use pkgs.autoAddDriverRunpath instead
warning: 'http://nixpkgs-cuda.cs-338.someonex.net:38180/nixpkgs-cuda' does not appear to be a binary cache
INFO:    gocryptfs not found, will not be able to use gocryptfs
WARNING: passwd file doesn't exist in container, not updating
WARNING: group file doesn't exist in container, not updating
Start
Runtime version: 12020
Driver version: 12050
Host memory initialized, copying to the device
Scheduled a cudaMemcpy, calling the kernel
Scheduled a kernel call
Max error: 0.000000

[SomeoneSerge]

I think this is backport-able, but I'll look into this some more

[github-actions]

Backport failed for release-24.05, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin release-24.05
git worktree add -d .worktree/backport-306730-to-release-24.05 origin/release-24.05
cd .worktree/backport-306730-to-release-24.05
git switch --create backport-306730-to-release-24.05
git cherry-pick -x 409cbbe61a551410d109b739a5cb0959a2c8db16 f6d9b4b6fc7f376e5f5ecacace951f57c155045c dbcf7cf697c601fc92da45453290b81587b87ef5 c3026ac986b6b21409a0240dcad4a411dad0d419

[SomeoneSerge]

Thank you again @ShamrockLee

[ShamrockLee]

Backport failed for release-24.05, because it was unable to cherry-pick the commit(s).

#306716 needs to be backported first so that we could backport this one.

[github-actions]

Successfully created backport PR for release-24.05:

• [Backport release-24.05] apptainer, singularity: precede system-level bin paths in defaultPath and fix singularity image running #324426