Tracking Issue: Prepare Python packages for PyPA build

[tjni]

This is a tracking issue that originates from #245509. I have been using that PR to reference work that is needed before merging it, but it distracts from the core logic in the PR itself, so I am going to use this issue to track the related work instead.

Prerequisites to merging

• python3.pkgs.flit-core: remove passthru tests #245671
• python3.pkgs.ninja-python: replace with a stub implementation  #246963
• python3.pkgs.ninja: export BIN_DIR constant #248826
• python3.pkgs.cmake: init stub at 3.26.4 #247064
• python3.pkgs.{cmake,ninja}: add __version__ to stub #249644
• python3.pkgs.scikit-build-core: add missing test dependencies #249646
• python3.pkgs.oldest-supported-numpy: init at 2023.8.3 #247073
• zeekscript: add build dependencies and fix metadata #246908
• python3.pkgs.mesonpep517: add build dependencies and fix metadata #246917
• python3.pkgs.hydra-core: fix tests with setuptools 67.5.0+ #246922
• python3.pkgs.d2to1: remove as it's archived and breaks with setuptools 68 #246829
• python3.pkgs.cx-freeze: switch to pyproject and rename #246900
• python3.pkgs.ciscoconfparse: fix build dependency metadata #246905
• python3.pkgs.astropy-extension-helpers: add build and test dependencies #246907
• python3.pkgs.rdflib: add pip test dependency #246909
• python3.pkgs.qcodes: add build and test dependencies and fix constraints #246912
• python3.pkgs.pytest-bdd: remove setuptools for compatibility #246915
• python3.pkgs.jupyter-packaging: fix tests with setuptools 67.5.0+ #246919
• python3.pkgs.josepy: fix tests with setuptools 67.5.0+ #246920
• python3.pkgs.django-extensions: add pip to test dependencies #246925
• python3.pkgs.ancp-bids: depend on wheel explicitly #246946
• python3.pkgs.aioslimproto: improve the derivation #247239
• python3.pkgs.ansi2html: dependency cleanup #247246
• python3.pkgs.aplpy: dependency cleanup #247250
• python3.pkgs.astroid: add wheel and pip dependencies #247258
• python3.pkgs.ax: add and move some build dependencies #247260
• python3.pkgs.botorch: add and move dependencies to nativeBuildInputs #247269
• python3.pkgs.bluetooth-data-tools: 1.6.1 -> 1.7.0 #247272
• python3.pkgs.clr-loader: add build dependencies  #247295
• python3.pkgs.cvxpy: add build dependencies and fix build constraints #247305
• python3.pkgs.datrie: add build dependencies and improve derivation #247306
• python3.pkgs.dbus-fast: build cython optimized version #247308
• python3.pkgs.aiohttp: fix tests with setuptools 67.5.0+ #247310
• python3.pkgs.jsonschema: add pip test dependency #247311
• python3.pkgs.virtualenv: add pip test dependency #247313
• python3.pkgs.bitlist: audit and update dependencies #247460
• python3.pkgs.broadbean: add build dependencies and fix constraints #247464
• python310Packages.pydeck: 0.7.1 -> 0.8.0 #247508
• python3.pkgs.xarray: replace setuptoolsBuildHook with normal dependencies #247852
• python3Packages.pytest-ansible: 3.1.5 -> 3.2.1 #246735
• python3.pkgs.wheel-filename: fix up build dependencies #247864
• python3.pkgs.xknx: fix up build dependencies #248067
• python3.pkgs.xknxproject: fix up build dependencies #248066
• python3.pkgs.snowflake-connector-python: fix build dependencies #248072
• python311Packages.hologram: pull patch to fix tests #245152
• python310Packages.asyncio-mqtt: rename file and add wheel dependency #248084
• python3.pkgs.mpmath: 1.2.1 -> 1.3.0 #248319
• python3.pkgs.diofant: 0.13.0 -> 0.14.0 #248511
• ablog: 0.10.33.post1 -> 0.11.14.post1 #248589
• macs2: 2.2.8 -> 2.2.9.1 #248595
• beautysh: patch "poetry" reference in pyproject.toml #248598
• brotab: patch out unnecessary "import pip" in setup.py #248601
• python3.pkgs.gevent: use cython3 and update build dependencies #248606
• python3.pkgs.scipy: relax build dependency constraints #248612
• cc2538-bsl: clean up dependencies and enable tests #248614
• python3.pkgs.pandas: add missing build dependencies #248623
• python3.pkgs.single-version: use poetry-core instead of poetry #248626
• python3.pkgs.dot2tex: fix duplicate script breaking installation #248640
• firefox_decrypt: add missing build dependencies #248643
• idasen: replace reference to poetry with poetry-core #248754
• python3.pkgs.zigpy: add build dependencies and fix constraints #248757
• python3.pkgs.statsmodels: add build dependencies and fix constraints #248759
• python3.pkgs.shapely: add missing build dependencies #248773
• python3.pkgs.shap: add missing build dependencies #248776
• python3.pkgs.liquidctl: add build dependencies #248780
• python3.pkgs.git-filter-repo: fix duplicate script error during install #248785
• python311Packages.mcstatus: 11.0.0 -> 11.0.1 #248787
• invoice2data: clean up some build and runtime dependencies #248798
• openrazer-daemon: do not install twice #248800
• play-with-mpv: unstable-2020-05-18 -> unstable-2021-04-02 #248822
• python3.pkgs.aws-lambda-builders: add missing pip test dependency #248836
• python3.pkgs.certbot-dns-*: ignore warning in setuptools 67.5.0+ #248840
• python3.pkgs.class-doc: patch out poetry dependency #248850
• python3.pkgs.insteon-frontend-home-assistant: unpin setuptools dependency #248859
• python3.pkgs.h5py: add missing build dependencies #248872
• python3.pkgs.socksio: unpin flit-core dependency #248943
• python3.pkgs.typer: unpin flit-core dependency #248946
• python3.pkgs.tiler: unpin setuptools-scm dependency #248949
• python3.pkgs.matplotlib: add missing build dependencies #249459
• python3.pkgs.doc8: fix build dependency constraints #249466
• steamback: add missing build dependencies #249473
• python3.pkgs.dask: remove references to versioneer #249486
• python3.pkgs.casa-formats-io: add missing build dependencies #249489
• python3.pkgs.graphql-relay: patch out setuptools dependency #249519
• python3.pkgs.awkward-cpp: add ninja build dependency #249648
• python3.pkgs.plux: 1.3.1 -> 1.4.0 #249763
• python3.pkgs.scramp: remove versioningit reference #249768
• python3.pkgs.libtmux: remove setuptools build dependency #249775
• python3.pkgs.levenshtein: update to cython 3 #249779
• python3.pkgs.pylint: unpin setuptools #249796
• awscli2: enable building without pip in the environment #249798
• python3.pkgs.versionfinder: add pip to test dependencies #249800
• python3.pkgs.schwifty: add missing build dependencies #249801
• python3.pkgs.pyproject-api: 1.5.0 -> 1.5.4 #249815
• aws-mfa: fix script duplication when installing with installer #249821
• corrscope: remove setuptools dependency #249837
• python3.pkgs.parts: 1.6.0 -> 1.7.0 #249838
• cve-bin-tool: add wheel dependency to pinned packaging #249842
• etesync-dav: pull patch into flask to work with setuptools 67.5.0+ #249847
• flatcam: do not pin setuptools in shapely #249849
• awsebcli: fixup, remove scripts from setup.py #249850
• python3.pkgs.python-swiftclient: patch out duplicate script #249854
• python3.pkgs.correctionlib: clean up / fix up build dependencies #249856
• python3.pkgs.vdirsyncer: add missing build dependencies #249916
• python3.pkgs.pg8000: add versioningit dependency #250036
• anki: remove pip from anki-build-python #250059
• python3.pkgs.pytest-test-utils: add missing build dependencies #250061
• faraday-agent-dispatcher: 2.4.0 -> 2.6.2 #250110
• ansible-doctor: use poetry-dynamic-versioning like upstream #250111
• ansible-later: use poetry-dynamic-versioning like upstream #250114
• python3.pkgs.python-rtmidi: 1.5.4 -> 1.5.5 #250117
• home-assistant: unpin and fix build dependencies #250124
• pipenv: remove wheel install test, enable unit tests #250153
• python3.pkgs.blosc2: add missing build dependencies #250155
• python3.pkgs.cffsubr: fix up build dependencies #250157
• python3.pkgs.compreffor: fix up build dependencies #250158
• python3.pkgs.pyprusalink: unpin setuptools dependency #250162
• python3.pkgs.pytest-mockservers: fully replace poetry with poetry-core #250165
• python3.pkgs.aiomysql: fix build dependency constraints #250206
• python3.pkgs.pyerfa: add missing build dependencies #250209
• python3.pkgs.ale-py: add missing ninja dependency #250211
• python3.pkgs.astropy: fix build dependencies #250215
• python3.pkgs.linear_operator: use setuptools-scm to get version #250240
• python3.pkgs.httpie: add pip as a test dependency #250256
• python3.pkgs.ml-dtypes: unpin build dependencies #250261 f13d239
• python3.pkgs.pyemd: add missing build dependencies #250262
• python3.pkgs.imageio-ffmpeg: patch out pip requirement #250458
• pypaInstallHook: enter "dist" directory before installing #250464
• python3.pkgs.cmsis-pack-manager: fix install using pypa installer #250649
• python3.pkgs.jupytext: relax build dependencies #250665
• anki: 2.1.61 -> 2.1.65 #229480
• matrix-synapse: relax build dependencies #250671
• python3.pkgs.u-msgpack-python: 2.7.2 -> 2.8.0 #250672
• python3.pkgs.pypiserver: add pip dependency and audit tests #250679
• python3.pkgs.afdko: add ninja build dependency #250681
• python3.pkgs.aioambient: clean up build dependency constraints #250687
• salt: fix duplicate scripts in wheel #250689
• python3.pkgs.fiona: relax build dependencies #250700
• python3.pkgs.jarowinkler: fix build dependencies #250706
• python3.pkgs.pyfibaro: 0.7.2 -> 0.7.3 #250798
• xonsh: add pip to test dependency and other changes #250801
• wapiti: 3.1.7 -> 3.1.8 #250814
• python3.pkgs.pulumi: add missing pip test dependency #250817
• python3.pkgs.aioflo: clean up build dependency constraints #250819
• python3.pkgs.pyopencl: add missing build dependencies #250822
• python3.pkgs.pvlib: add missing build dependencies #250823
• python3.pkgs.python-lsp-server: relax jedi dependency constraint #250824
• Remove some unnecessary dependencies on setuptools, wheel, and poetry #250850
• Relax or remove jupyterlab from some packages #250886
• Remaining Python bootstrapping fixes (part 1) #250905
• Remaining Python bootstrapping fixes (part 2) #251047
• python310Packages.esphome-dashboard-api 669bfe2
• python310Packages.remarshal abc57e2
• python310Packages.distributed 3d3b361
• python310Packages.fastparquet 2ead800
• python311Packages.hahomematic: 2023.7.3 -> 2023.8.2  #247885
• python3.pkgs.monai-deploy: replace versioneer-518 with versioneer #251065
• python3.pkgs.python-matter-server: relax setuptools dependency #251096
• Fix some more issues on staging-next related to Python bootstrap changes #251240
• Fix more staging-next Python failures #251533
• python2Packages.ninja: alias it to ninja build tool #251548
• python310Packages.fountains: remove patch leftover from merge #251677
• python310Packages.ansible-pylibssh: patch out setuptools_scm_git_archive #251679
• python310Packages.scikit-image: relax numpy dependency constraint #251708
• molecule: add missing build dependencies #251730
• python310Packages.fe25519: remove patch leftover from merge #251733
• python310Packages.opentsne: add missing build dependencies #251736
• python310Packages.vega: fully switch to poetry-core #251739

Makes life easier

• python: update maintainers for some core python packages #246926
• python3.pkgs.hydra-core: unbreak on darwin #247207
• treewide: fix some builds in darwin sandbox #246723
• python3.pkgs.setuptools-git-versioning: init at 1.13.5 #248019
• python3.pkgs.orjson: 3.9.2 -> 3.9.4 #248273
• python3.pkgs.protobuf: fix build after updating protobuf to 3.23.4 #248231
• gupnp, gupnp_1_6: fix compilation #246931
• python3.pkgs.pyqt-builder: 1.14.1 -> 1.15.2 #248760
• python310Packages.poetry-core: 1.5.1 -> 1.6.1 #244535
• cobang: minor fix to patch replacing poetry with poetry-core #249841

[tjni]

There are still a few lingering packages that need to be fixed, but the main body of the changes have been merged back into master and will be fixed one-by-one when time permits. I'll be closing this ticket now. I want to thank everyone who has been so helpful with reviewing the PRs here.