Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(bzlmod): set the default_version to the python_version flag #2253

Merged
merged 19 commits into from
Oct 7, 2024

Conversation

aignas
Copy link
Collaborator

@aignas aignas commented Sep 26, 2024

With this change we set the default value of --python_version when
the python.toolchain is used in bzlmod and we generate the
appropriate config settings based on the registered toolchains and
given overrides by the root module.

This means that we expect the --python_version to be always set to
a non-empty value under bzlmod and we can cleanup code which was
handling //conditions:default case. This also means that we can
in theory drop the requirement for python_version in pip.parse
and just setup dependencies for all packages that we find in the
requirements.txt file and move on. This is left as future work
by myself or anyone willing to contribute. We can also start reusing
the same whl_library instance for multi-platform packages because
the python version flag is always set - this will simplify the layout
and makes the feature non-experimental anymore under bzlmod.

Summary:

  • Add @pythons_hub to the WORKSPACE with dummy data to make
    pythons_hub work.
  • Add MINOR_MAPPING and PYTHON_VERSIONS to pythons_hub to
    generate the config settings.
  • Remove handling of the default version in pypi code under bzlmod.

Work towards #2081, #260, #1708

Summary:
* Add pythons_hub to the WORKSPACE with dummy data to make
  pythons_hub work.
* Add MINOR_MAPPING and PYTHON_VERSIONS to pythons_hub to
  generate the config settings.
* Remove the default_version usage from pythons_hub, this
  will be important when consolidating the code later. We can
  also potentially relax the need for requiring users to set
  python_version attribute because python_versions can now be
  just read from the interpreters.bzl file and we can just setup
  the python deps for all available interpreters instead. This
  also makes sharing the same whl repo for multiple python versions
  work as the deps in the whl_library will always resolve correctly
* Update the test code to continue to work
@aignas aignas marked this pull request as ready for review September 26, 2024 10:58
if not BZLMOD_ENABLED:
hub_repo(
name = "pythons_hub",
minor_mapping = {},
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we wanted, we could do extra plumbing within the pythons_hub and pass the correct values for minor_mapping and python_versions from the //python:versions.bzl, but given that WORKSPACE is on the way out and this is temporary, I thought it would be best to leave it like this.

@@ -528,7 +529,7 @@ def config_settings_test_suite(name): # buildifier: disable=function-docstring

config_settings(
name = "dummy",
python_versions = ["3.8", "3.9", "3.10"],
python_versions = ["3.7", "3.8", "3.9", "3.10"],
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
python_versions = ["3.7", "3.8", "3.9", "3.10"],
# NOTE: We don't have any `3.7` in `//python:versions.bzl` and
# this value has been selected on purpose to test what happens if the value
# is unknown to rules_python - i.e. we should still be able to work.
python_versions = ["3.7", "3.8", "3.9", "3.10"],

aignas added a commit to aignas/rules_python that referenced this pull request Sep 27, 2024
However this is not a test for the code in question as those tests
probably need to wait for bazelbuild#2253 to land to make the test writing easier.
@rickeylev
Copy link
Collaborator

This also means that we can in theory drop the requirement for python_version in pip.parse

Oh? How? Aren't we still stuck with the case where the same python version interpreter has to go find and download something? sdists?

I think I see what you mean for a pure-wheel build. Well, maybe. When requirements.in is resolved by e.g. compile_pip_requirements to a particular python-specific requirements.txt, the python version has, effectively, been decided. But -- I don't recall requirements.txt storing the python version? So how do we know that foo == 1.2.3 # hash blabla means el.g. foo-1.2.3-cpython-313.tar.gz (as opposed to foo-1.2.3-cpython-310.tar.gz) ?

internal_setup.bzl Outdated Show resolved Hide resolved
python/private/pypi/config_settings.bzl Outdated Show resolved Hide resolved
name = is_python,
actual = Label("//python/config_settings:" + is_python),
flag_values = {
Label("//python/config_settings:_python_version_major_minor"): python_version,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, maybe we should expose python_version_major_minor as a public setting. Otherwise, users don't have a good way to perform e.g. select({"3.10": ..., "3.11": ...}), which is the most common type of selection on python version.

Don't have to address in this PR though.

…o feat/default-version-python-hub

Resolve changelog conflict
@rickeylev
Copy link
Collaborator

CI's error is about running it under Bazel 6 -- I think it's trying to say pythons_hub isn't defined. Though, the (Mac, pystar, workspace) job is also failing and that is using Bazel 7. Looks like something to do with workspace builds.

@rickeylev
Copy link
Collaborator

Ok, think I found the problem (wrong load in a test file). Pushed a fix. CI is acting weird right now, though. Maybe it'll be in a better mood when you log on; I suggest pushing a no-op change and having it start over.

@aignas
Copy link
Collaborator Author

aignas commented Oct 7, 2024

This also means that we can in theory drop the requirement for python_version in pip.parse

Oh? How? Aren't we still stuck with the case where the same python version interpreter has to go find and download something? sdists?

I think I see what you mean for a pure-wheel build. Well, maybe. When requirements.in is resolved by e.g. compile_pip_requirements to a particular python-specific requirements.txt, the python version has, effectively, been decided. But -- I don't recall requirements.txt storing the python version? So how do we know that foo == 1.2.3 # hash blabla means el.g. foo-1.2.3-cpython-313.tar.gz (as opposed to foo-1.2.3-cpython-310.tar.gz) ?

The idea is that we can just always define whl_library instances for all of the dists we find in a requirements file and we don't need to know the version of the python interpreter that we are getting. Sometimes the wheels could be missing though, but in the issue linked to the PR (#1708), the same requirements.txt is used for each python_version, so sometimes requirements.txt can be multi-version compatible.

This is only true for wheels though. For sdists we would need to iterate through registered python toolchain repository minor versions and add a whl_library instance per python version for each sdist we find. This is until we can build sdist to whl in a build context.

@aignas aignas enabled auto-merge October 7, 2024 01:48
@aignas aignas added this pull request to the merge queue Oct 7, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 7, 2024
@aignas aignas added this pull request to the merge queue Oct 7, 2024
Merged via the queue into bazelbuild:main with commit 33fa845 Oct 7, 2024
4 checks passed
aignas added a commit to aignas/rules_python that referenced this pull request Oct 7, 2024
…major_minor

With this change the users can simply reuse our internal flag that will
correctly report the `X.Y` version in `select` statements. If users
previously depended on now removed `is_python_config_setting` now they
have an alternative.

Followup to bazelbuild#2253
github-merge-queue bot pushed a commit that referenced this pull request Oct 7, 2024
…major_minor (#2275)

With this change the users can simply reuse our internal flag that will
correctly report the `X.Y` version in `select` statements. If users
previously depended on now removed `is_python_config_setting` now they
have an alternative.

Followup to #2253
@aignas aignas deleted the feat/default-version-python-hub branch October 7, 2024 06:05
aignas added a commit to aignas/rules_python that referenced this pull request Oct 26, 2024
…ble versions

I think that bazelbuild#2253 has introduced an issue that went unnoticed because
we did not have an integration test checking this. The main issue is
that config settings of `is_python_3.x.y` were being generated only for
registered toolchains and that is not necessarily the desired behaviour
-- we may want to use the config settings for more values, so coupling
that was a mistake.
github-merge-queue bot pushed a commit that referenced this pull request Oct 26, 2024
…#2350)

#2253 broke how the config settings are generated and only generated the
config
setting values for the python version values that we would have the
registered
toolchains for. This PR restores the previous behaviour. However, if the
root
module uses `python.override` to remove the allowed toolchains, then
`config_settings` will be also affected.
aignas added a commit that referenced this pull request Oct 27, 2024
…#2350)

PR #2253 broke how the config settings are generated and only generated the
config
setting values for the python version values that we would have the
registered
toolchains for. This PR restores the previous behaviour. However, if the
root
module uses `python.override` to remove the allowed toolchains, then
`config_settings` will be also affected.

(cherry picked from commit 9340a81)

Conflicts:
- CHANGELOG.md
aignas added a commit that referenced this pull request Oct 27, 2024
…#2350)

PR #2253 broke how the config settings are generated and only generated the
config
setting values for the python version values that we would have the
registered
toolchains for. This PR restores the previous behaviour. However, if the
root
module uses `python.override` to remove the allowed toolchains, then
`config_settings` will be also affected.

(cherry picked from commit 9340a81)

Conflicts:
- CHANGELOG.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants