python3.pkgs.ninja-python: replace with a stub implementation

[tjni]

Description of changes

This maps to the ninja package on PyPI, which downloads ninja from the web. This was packaged in #243967.

@999eagle, @SuperSandro2000 I would like to replace that implementation with a stub that directly uses ninja in nixpkgs. This capability was requested by @FRidh in scikit-build/ninja-python-distributions#127 – however upstream would like to delegate this capability to PEP 517 backends.

Only certain backends support this out-of-the-box, such as scikit-build-core and (maybe) meson-python. Notably, some projects are still using the older scikit-build, which does not support this. This impacts #245509 because the way I have set up build does dependency validation and those projects either need to include this ninja package (picking up its version of ninja) or we need to patch their pyproject.toml.

Hence, I'd like to stub out this package and converge on using just the single ninja program provided in nixpkgs.

If this sounds reasonable, I'd also like some opinions on whether it's worth changing the name of this package to just ninja. On the one hand, duplicate names can be confusing. On the other hand, it could also be confusing to someone building a Python package and reading the pyproject.toml that they need to use ninja-python instead of simply ninja.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[tjni]

I also went ahead and optimistically renamed ninja-python to ninja so it can be a transparent shim to Python packages. My sense is that is how most Python packages receive ninja and, if this works, it will require less work to get dependency validation passing in the new Python bootstrapping process.

[tjni]

I think I figured out how to do this properly. We don't need to propagate the original tool in the PATH. We only need to:

1. Define entry points that call into the stubs which wrap the tools. I didn't realize initially that this was how it worked.
2. Inherit the setup hooks from the parent package.

[bryango]

I believe python2 is broken by this PR; it lands on master via #248496, and I can get python2 to build on its parent but afterward, not anymore. Maybe it's time to give up python2 and mark it as broken...

[tjni]

Can you share the command you're running? I will try to build Python 2 in a little bit, but the reason I ask is because this doesn't seem connected to Python 2 itself but perhaps some package that is using it? Were you building nixops and it passed before but fails after?

[FRidh]

I believe python2 is broken by this PR; it lands on master via #248496, and I can get python2 to build on its parent but afterward, not anymore. Maybe it's time to give up python2 and mark it as broken...

It builds fine on both master and staging-next.

[bryango]

Hey sorry for the lack of details; the direct failure is always from flit-core; I was not building nixops myself but actually gimp with python2, and observe almost identical errors; build log excerpt:

error: builder for '/nix/store/1px104l2nbjb0v88fzcqdk21w2l0k1cq-python2.7-flit-core-3.9.0.drv' failed with exit code 2;
       last 10 log lines:
       >     backend=self.pep517_backend,
       >   File "/nix/store/vm2917jk0jsbanpkl9z74fflz1achliq-python2.7-pip-20.3.4/lib/python2.7/site-packages/pip/_internal/operations/build/metadata.py", line 35, in generate_metadata
       >     metadata_dir
       >   File "/nix/store/vm2917jk0jsbanpkl9z74fflz1achliq-python2.7-pip-20.3.4/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 196, in prepare_metadata_for_build_wheel
       >     '_allow_fallback': _allow_fallback,
       >   File "/nix/store/vm2917jk0jsbanpkl9z74fflz1achliq-python2.7-pip-20.3.4/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 284, in _call_hook
       >     raise BackendUnavailable(data.get('traceback', ''))
       > BackendUnavailable
       > Removed file:///build/source/flit_core from build tracker '/build/pip-req-tracker-M2uOIa'
       > Removed build tracker: '/build/pip-req-tracker-M2uOIa'
       For full logs, run 'nix log /nix/store/1px104l2nbjb0v88fzcqdk21w2l0k1cq-python2.7-flit-core-3.9.0.drv'.
error: 1 dependencies of derivation '/nix/store/xc2v165ls7kpp1ghds77qr9jirikalf0-python2.7-ninja-1.11.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/6pzzyhf0q3hi8im0pbq2l9mzvc05x9n5-python2.7-pycairo-1.18.2.drv' failed to build
error: 1 dependencies of derivation '/nix/store/w7c5pp51sax64b9b8j8xd9272l8z73pl-python-2.7.18.6-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/cci2flclhx9fw9zb3afdkg5j4r1b78zj-gimp-2.10.34.drv' failed to build
error: 1 dependencies of derivation '/nix/store/n0kvgsdzjm4p1x0ll7is5ax84rnqhicr-user-drv-overlays.drv' failed to build

However, the commit history of flit-core is rather benign (only some tests get removed), and I cannot figure out why it failed; after traversing the dependency tree it seems to me that this commit is the only major change that might contribute to the failure. I am no expert in python though; so this might be wrong...

[bryango]

I believe python2 is broken by this PR; it lands on master via #248496, and I can get python2 to build on its parent but afterward, not anymore. Maybe it's time to give up python2 and mark it as broken...

It builds fine on both master and staging-next.

Ah my bad, it should be python2.7-flit-core which is a dependency of python-2.7.18.6-env, I missed the -env part... Sorry for the misleading information... Now it seems to me that the culprit lies somewhere else; sorry for the inconvenience.

[tjni]

This makes sense to me! I'll see what I can do to fix it.

[bryango]

I think I roughly understand what happened now: the only reason that reverting this PR works for me is because here flit-core is introduced as a build dependency. It may be that python2.7-flit-core has been failing forever, but I wouldn't notice that because it never entered my dependency tree before. So yeah, the culprit should be flit-core all along, but I dismissed it too early based on its recent commit history... ninja-python should be fine on its own!

[tjni]

I opened #251548 to undo this change on Python 2 and buy myself some time to think about how to make this better. I'm sorry about the breakage.

[bryango]

Wow, awesome! Thank you very much for the help!