Python: change bootstrapping. #248866

FRidh · 2023-08-13T08:38:14Z

Description of changes

This is #245509 but then on python-updates branch so we can use Hydra.

Things done

tjni · 2023-08-16T03:38:02Z

I have rebased this on staging-next for now to make it a little easier to reason about for me. I propose:

For commits that are merged into master, pick it up after it's been merged into staging-next.
For commits that are merged into staging, cherry-pick over to here and when it comes time to rebase this onto staging, it should be de-duplicated.

FRidh · 2023-08-17T06:34:36Z

I'm going to start one more run. I'm pretty confident we can merge then. Please don't add any more unrelated updates, only (larger) fixes.

FRidh · 2023-08-17T12:14:59Z

jedi has one failing test on all platforms which has to be resolved

tjni · 2023-08-17T15:48:31Z

jedi needed to be updated to 0.19.0 to be compatible with attrs 23.1.0

tjni · 2023-08-18T02:03:20Z

The last 3 commits are PRs that were either merged or are targeting master but they should unblock testing of a few hundred dependent packages all together.

FRidh · 2023-08-18T06:23:00Z

pg8000 will need to be fixed

adding license file 'LICENSE'
writing manifest file 'pg8000.egg-info/SOURCES.txt'

ERROR Missing dependencies:
        versioningit>=2.1.0

After that we can trigger hydra again.

doronbehar · 2023-08-18T09:33:35Z

Small scipy update at #249921 .

FRidh · 2023-08-19T11:14:26Z

About 4000 failures on this branch.

tjni · 2023-08-19T21:57:24Z

I looked through a good number of the failures and listed them at the top of #247287. The biggest improvement will be from fixing home-assistant packages (#250124). I'm not entirely sure where else we can get big improvements. I'd like to merge the outstanding PRs, merge those changes in master into staging-next, and then rebase on top again to get an up to date list of broken packages.

anund · 2023-08-20T04:56:55Z

pkgs/development/python-modules/pip/default.nix

+    installShellCompletion --cmd pip \
+      --bash <($out/bin/pip completion --bash) \
+      --fish <($out/bin/pip completion --fish) \
+      --zsh <($out/bin/pip completion --zsh)


<($out/bin/pip completion --zsh --no-cache-dir) will properly clear the warning about .cache/pip not being writable. zsh may not work until pypa/pip#12166 is closed. (given pip is at the bottom of ~40000 packages leaving it in seems fine)

Also fixes #224610

Thank you for figuring that out. This was merged without that change, but we should add this in a separate commit.

Please do. I'm working on another change atm and don't need the two things coupled.

Tacked it on to this change https://github.com/NixOS/nixpkgs/pull/250398/files

Thank you, I appreciate you creating that PR. I'm sorry I hadn't found the time yet today.

After #248866, we have stronger build-time dependency validation This fixes "ERROR: Missing dependencies: ninja"

SuperSandro2000 · 2023-08-27T18:46:55Z

pkgs/development/python-modules/scramp/default.nix

+    sed -i "/versioningit >=/d" pyproject.toml
    sed -i '/^name =.*/a version = "${version}"' pyproject.toml
    sed -i "/dynamic =/d" pyproject.toml


nit: those could have been combined into one command

Yes, you're right. At the same time, even though it's less efficient, I feel that keeping separate commands here is more readable. But this motivates me to look into whether we can handle versioningit better in general, thank you!

I always find less code overhault more readable unless you are using some special character shortcuts. Like here, eg deduplicating pyproject.toml but as said it is not that super important and focusing on getting a setup-hook for verioningit is a better solution anyway.

nixos-discourse · 2023-08-28T12:35:48Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/psa-poetry2nix-is-currently-broken-with-nixpkgs-unstable/32281/1

In NixOS/nixpkgs#248866, `pip` stopped getting included by default when building packages.

jqqqqqqqqqq · 2023-10-28T07:40:11Z

Should we backport this to 23.05?

I'm seeing errors on 23.05 because some inconsistency happened:

[ERROR]   stderr) error: anonymous function at /nix/store/r5b4xjc4gvraxdslr98f1qpn62vp3gw8-source/pkgs/development/python-modules/wheel/default.nix:1:1 called with unexpected argument 'flit-core'
[ERROR]   stderr)
[ERROR]   stderr)        at /nix/store/r5b4xjc4gvraxdslr98f1qpn62vp3gw8-source/lib/customisation.nix:80:16:
[ERROR]   stderr)
[ERROR]   stderr)            79|     let
[ERROR]   stderr)            80|       result = f origArgs;
[ERROR]   stderr)              |                ^
[ERROR]   stderr)            81|
[ERROR]   stderr) (use '--show-trace' to show detailed location information)
[ERROR]  failure) Child process exited with error code: 1

And 23.05 is still using the old wheel package:

cat /nix/store/r5b4xjc4gvraxdslr98f1qpn62vp3gw8-source/pkgs/development/python-modules/wheel/default.nix
{ lib
, buildPythonPackage
, fetchFromGitHub
, bootstrapped-pip
, setuptools
}:

buildPythonPackage rec {
  pname = "wheel";
  version = "0.38.4";
  format = "other";

  src = fetchFromGitHub {
    owner = "pypa";
    repo = pname;
    rev = version;
    hash = "sha256-yZLU0t/nz6kfnnoLL15bybOxN4+SJUaTJsCpGffl1QU=";
    name = "${pname}-${version}-source";
    postFetch = ''
...

kirillrdy · 2023-10-28T07:44:17Z

Should we backport this to 23.05?

I'm seeing errors on 23.05 because some inconsistency happened:

[ERROR]   stderr) error: anonymous function at /nix/store/r5b4xjc4gvraxdslr98f1qpn62vp3gw8-source/pkgs/development/python-modules/wheel/default.nix:1:1 called with unexpected argument 'flit-core'
[ERROR]   stderr)
[ERROR]   stderr)        at /nix/store/r5b4xjc4gvraxdslr98f1qpn62vp3gw8-source/lib/customisation.nix:80:16:
[ERROR]   stderr)
[ERROR]   stderr)            79|     let
[ERROR]   stderr)            80|       result = f origArgs;
[ERROR]   stderr)              |                ^
[ERROR]   stderr)            81|
[ERROR]   stderr) (use '--show-trace' to show detailed location information)
[ERROR]  failure) Child process exited with error code: 1

And 23.05 is still using the old wheel package:

cat /nix/store/r5b4xjc4gvraxdslr98f1qpn62vp3gw8-source/pkgs/development/python-modules/wheel/default.nix
{ lib
, buildPythonPackage
, fetchFromGitHub
, bootstrapped-pip
, setuptools
}:

buildPythonPackage rec {
  pname = "wheel";
  version = "0.38.4";
  format = "other";

  src = fetchFromGitHub {
    owner = "pypa";
    repo = pname;
    rev = version;
    hash = "sha256-yZLU0t/nz6kfnnoLL15bybOxN4+SJUaTJsCpGffl1QU=";
    name = "${pname}-${version}-source";
    postFetch = ''
...

I don't think it fits backporting criteria, also 23.11 is imminent

FRidh requested review from jonringer and adisbladis as code owners August 13, 2023 08:38

github-actions bot added the 6.topic: python label Aug 13, 2023

FRidh mentioned this pull request Aug 13, 2023

Switch to PyPA build, installer, and flit-core to build Python packages #245509

Closed

12 tasks

ofborg bot added 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Aug 13, 2023

tjni marked this pull request as draft August 16, 2023 02:58

tjni changed the base branch from staging to staging-next August 16, 2023 03:05

tjni force-pushed the python-updates branch from f73e66b to 574c149 Compare August 16, 2023 03:24

tjni force-pushed the python-updates branch 7 times, most recently from 866cd47 to fbed5e9 Compare August 17, 2023 06:03

anund reviewed Aug 20, 2023

View reviewed changes

yu-re-ka pushed a commit that referenced this pull request Aug 23, 2023

python3.pkgs.afdko: add ninja to nativeBuildInputs (#250987)

1e23ff5

After #248866, we have stronger build-time dependency validation This fixes "ERROR: Missing dependencies: ninja"

SuperSandro2000 reviewed Aug 27, 2023

View reviewed changes

Artturin mentioned this pull request Aug 28, 2023

grapejuice: fix pip install #252054

Merged

12 tasks

doronbehar mentioned this pull request Aug 29, 2023

antlr4_13: init at 4.13.0 and set as default #238543

Merged

12 tasks

jfly added a commit to jfly/snow that referenced this pull request Aug 29, 2023

Add pip to shtuff's nativeCheckInputs

9d09d37

In NixOS/nixpkgs#248866, `pip` stopped getting included by default when building packages.

This was referenced Aug 30, 2023

python310Packages.heudiconv: patch out versioningit build dependency #252264

Closed

python310Packages.recordlinkage: add missing build dependencies #252342

Closed

python310Packages.pdf2docx: add pip as a build dependency #252361

Merged

KiruyaMomochi mentioned this pull request Sep 1, 2023

Segmentation fault nix-community/poetry2nix#1291

Closed

bbjubjub2494 mentioned this pull request Sep 3, 2023

Python dependencies are no longer checked #253131

Closed

yajo mentioned this pull request Sep 5, 2023

python.pipInstallHook: avoid producing wrong direct_url.json file #232414

Open

12 tasks

DavHau mentioned this pull request Sep 11, 2023

python3.pkgs.pypaBuildHook: fix conflicts via propagated inputs #254547

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python: change bootstrapping. #248866

Python: change bootstrapping. #248866

FRidh commented Aug 13, 2023 •

edited

Loading

tjni commented Aug 16, 2023

FRidh commented Aug 17, 2023

FRidh commented Aug 17, 2023

tjni commented Aug 17, 2023

tjni commented Aug 18, 2023

FRidh commented Aug 18, 2023

doronbehar commented Aug 18, 2023

FRidh commented Aug 19, 2023

tjni commented Aug 19, 2023

anund Aug 20, 2023

anund Aug 20, 2023

tjni Aug 20, 2023

anund Aug 21, 2023

anund Aug 22, 2023

tjni Aug 22, 2023

SuperSandro2000 Aug 27, 2023

tjni Aug 27, 2023

SuperSandro2000 Aug 28, 2023

nixos-discourse commented Aug 28, 2023

jqqqqqqqqqq commented Oct 28, 2023

kirillrdy commented Oct 28, 2023

Python: change bootstrapping. #248866

Python: change bootstrapping. #248866

Conversation

FRidh commented Aug 13, 2023 • edited Loading

Description of changes

Things done

tjni commented Aug 16, 2023

FRidh commented Aug 17, 2023

FRidh commented Aug 17, 2023

tjni commented Aug 17, 2023

tjni commented Aug 18, 2023

FRidh commented Aug 18, 2023

doronbehar commented Aug 18, 2023

FRidh commented Aug 19, 2023

tjni commented Aug 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nixos-discourse commented Aug 28, 2023

jqqqqqqqqqq commented Oct 28, 2023

kirillrdy commented Oct 28, 2023

FRidh commented Aug 13, 2023 •

edited

Loading