Poetry build is naming the wheel incorrectly

[alexifm]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• macOS 10.15.7

• 1.1.4

• poetry-core 1.0.0

Issue

This is a spin off of #3508 as further investigation revealed there were two problems uncovered. This is about the naming of the wheels when using poetry build.

Poetry is giving the wrong version tag to the built wheel. The following is the output of Poetry build:

$ poetry build 
Building build_test (0.1.0)
  - Building sdist
  - Built build_test-0.1.0.tar.gz
  - Building wheel
A setup.py file already exists. Using it.
running build
running build_py
  - Built build_test-0.1.0-cp39-cp39-macosx_10_15_x86_64.whl

• Note that the Py version is 3.9 but below, the project is shown to be exclusively 3.8. In fact, the only place where Python 3.9 is installed on my system is either through brew as a dependency for something or through the pipx isolated environment used to install Poetry.
• Even though the build script does nothing, and even if there was something to do, the build would be executed in the correct venv (I've verified this) meaning that the issue really is just with the name of the wheel.

Here is how poetry is installed:

$ pipx list
venvs are in /Users/alexifm/.local/pipx/venvs
apps are exposed on your $PATH at /Users/alexifm/.local/bin
   package poetry 1.1.4, Python 3.9.1
    - poetry

Poetry config:

$ poetry config --list
cache-dir = "/Users/alexifm/Library/Caches/pypoetry"
experimental.new-installer = true
installer.parallel = true
virtualenvs.create = false
virtualenvs.in-project = null
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/alexifm/Library/Caches/pypoetry/virtualenvs

And here is the project definition that requires a build (note that the project is exclusively Python 3.8).

Project virtual environment:

$ pyenv versions
  system
  3.8.5
  3.8.5/envs/build_test
* build_test (set by /Users/alexifm/Projects/build_test/.python-version)

pyproject.toml:

[tool.poetry]
name = "build_test"
version = "0.1.0"
description = ""
authors = ["Alex Papanicolaou <[email protected]>"]
build = "build.py"

[tool.poetry.dependencies]
python = "~3.8"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry_core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

build.py:

"""No build."""

def build(setup_kwargs):
    """Build."""
    pass

Probable Reason

poetry-core generates the information for the wheel tag here:
https://github.com/python-poetry/poetry-core/blob/master/poetry/core/masonry/builders/wheel.py#L244

The vendored packaged packaging produces the tags here:
https://github.com/python-poetry/poetry-core/blob/master/poetry/core/_vendor/packaging/tags.py#L744

Finally, it looks like since no kwargs are passed to sys_tags, then the argument python_version isn't passed to cpython_tags and it defaults to the system Python, ie whatever is running Poetry.
https://github.com/python-poetry/poetry-core/blob/master/poetry/core/_vendor/packaging/tags.py#L234

[alexifm]

Follow-up: the function sys_tags in packaging calls the function interpreter_name linked below which pulls the interpreter name of the Python from the sys module, ie the one that is running Poetry. I doubt it's a priority for Poetry to support other Python interpreters but this basically means that everything has to be CPython.

https://github.com/python-poetry/poetry-core/blob/abe9fe59fd7a0b181463461273c16af8a258a969/poetry/core/_vendor/packaging/tags.py#L696

[alexifm]

@abn We've talked previously about an issue around poetry build. I believe you may have indicated that this particular use case of custom build scripts (ie. for Cython) is not really supported so this issue may be a bit of a dead end?

[alexifm]

Okay, I figured out one issue.

If a user has activated their own venv with a tool like pyenv, set virtualenvs.create to false, but also has an environment in {cache-dir}/virtualenvs/envs.toml, the venv detected will be incorrect. This setup can happen if you don't set things up perfectly initially and there's a lingering listed environment in the cache.

The problem comes from this conditional:
https://github.com/python-poetry/poetry/blob/1.1.4/poetry/utils/env.py#L463

Here's what happens

1. A poetry virtualenv is checked for here: https://github.com/python-poetry/poetry/blob/1.1.4/poetry/utils/env.py#L446-L450
2. If you have created a virtual environment with poetry already then it'll show up in the envs.toml file and the variable env is set on this line:

   poetry/poetry/utils/env.py

   Line 450 in 8312e3f

   env = envs.get(base_env_name)

3. If another venv is activated, say with pyenv, that'll be picked up here:

   poetry/poetry/utils/env.py

   Line 457 in 8312e3f

   env_prefix = os.environ.get("VIRTUAL_ENV", os.environ.get("CONDA_PREFIX"))

4. Following Add support for git dependencies #2, this conditional will evaluate True.

   poetry/poetry/utils/env.py

   Line 463 in 8312e3f

   if not in_venv or env is not None:

5. If you've configured Poetry to not create virtualenvs, then this will evaluate True and return the system python.

   poetry/poetry/utils/env.py

   Line 473 in 8312e3f

   if not create_venv:

Two solutions:

• the user needs to clean up the envs.toml file.
• The logic in .get needs adjustment so that it actually uses the environment the user has activated.

[alexifm]

Figured out the problem: even after fixing the issue of the blocked venv in the previous comment, the name of the wheel ends up incorrect (it gets labeled 3.9 despite being built by a 3.8 venv). It comes from the function sys_tags in packaging. sys_tags gives the tags for the python for sys.executable, ie the Python that is orchestrating the build and not the one actually building the wheel. We need the tags for the Python that builds the wheel.

To fix this, I hacked a solution where I pass the environment to the builder classes in poetry.core.masonry.builders instead of just the path to the executable. That is, I tweaked it so I pass self.env instead of self.env.python here:
https://github.com/python-poetry/poetry/blob/master/poetry/console/commands/build.py#L35-L36

Then, instead of using sys_tags I use cpython_tags(python_version=self.env.get_version_info()):
https://github.com/python-poetry/poetry-core/blob/master/poetry/core/masonry/builders/wheel.py#L244

This ends up working correctly.

[sinoroc]

It might be worth clarifying:

• What are you using exactly to create virtual environments? Since as far as I know contrary to what you are saying here, pyenv is not able to create virtual environments.
• Are you using pyenv-virtualenv? It seems it has not seen updates in quite a long time, so it might not be creating virtual environments that are standards conform or something else that causes issues to poetry.
• Is it possible to recreate the issue with virtual environments created by other (more standard conform) tools such as virtualenv, or Python's standard library own venv, or tox, etc.?

[alexifm]

I just replicated using Poetry's own virtual environments so it's not a pyenv thing.

$ poetry build -vvv                     
Using virtualenv: /Users/alexifm/Library/Caches/pypoetry/virtualenvs/build-test-9tgLzTmA-py3.8   # <- Poetry's Py3.8 environment
Building build_test (0.1.0)
  - Building sdist
  - Adding: /Users/alexifm/Projects/build_test/build_test/__init__.py
  - Adding: pyproject.toml
  - Built build_test-0.1.0.tar.gz
  - Building wheel
  - Adding: /Users/alexifm/Projects/build_test/build_test/__init__.py
running build
running build_py
Skipping: /Users/alexifm/Projects/build_test/COPYING
Skipping: /Users/alexifm/Projects/build_test/LICENSE
  - Built build_test-0.1.0-cp39-cp39-macosx_10_15_x86_64.whl   # <- Named with 3.9 used to run Poetry (pipx)

ETA: It's also not a pipx thing as I can replicate it with Poetry installed through homebrew.

[matzhaugen]

I'm confused about two things here:

1. In the pyproject.toml file, why is the build field in the [tool.poetry] section not documented in the docs?
2. It seems that if the build field is missing from the pyproject file the wheel will always have a none-any.whl suffix, as seen here . Is this intentional? This means unless you put a build field in there will never be more specific wheel than none-any. It seems that the wheel name should be independent on whether a build script is present or not, no? E.g. If I add a setup.py file with a different version that what I specify in the pyproject.toml there will be a redundancy, and possible conflict.

I'm not sure which platforms use the any-none wheel specifications. Maybe there is something I'm missing.

[alexifm]

1. In the `pyproject.toml` file, why is the `build` field in the `[tool.poetry]` section not documented in the docs?

It's an unsupported feature and it's why this issue is not too big of a priority.

2. It seems that if the `build` field is missing from the pyproject file the wheel will always have a `none-any.whl` suffix, as [seen here ](https://github.com/python-poetry/poetry-core/blob/master/poetry/core/masonry/builders/wheel.py#L243-L255). Is this intentional? This means unless you put a `build` field in there will never be more specific wheel than `none-any`. 

Basically, if you're not building anything, then the package is platform independent and you don't need a fancy wheel name. If you need to build something, say Cython, then the wheel name has to account for the Python version and the platform that did the compiling.

[gitonthescene]

Seems the same issue as #4168.

[gitonthescene]

Linking related issue. #2613

[tedivm]

Basically, if you're not building anything, then the package is platform independent and you don't need a fancy wheel name. If you need to build something, say Cython, then the wheel name has to account for the Python version and the platform that did the compiling.

This assumption is fine as a default but doesn't hold for all use cases. There are cases where the build script is used to create asset files that are needed by the package but which are not platform specific (for example, building a json file for data that's going into the package). Having an option to explicitly say "this is an platform independent package" would be a lot more flexible than having the program try to guess what developers are attempting to do.

[jrandall]

I also have this issue and note that it worked in poetry 1.0.9 but was broken when we moved to 1.1.x

[tedivm]

Here's a script I run after building with poetry to fix the modules. Note that this only works properly if you are using a pure python module- if you are compiling libraries and shipping them then you should not run this script.

#!/usr/bin/env bash
set -e

ORIGINAL_PACKAGE=$1
START_DIR=$(pwd)
TEMP_DIR=$(mktemp -d)
WORKING_WHEEL=$TEMP_DIR/package.whl
FINAL_PACKAGE=$(echo $ORIGINAL_PACKAGE | rev | cut -d"-" -f4- | rev)-py3-none-any.whl

if [[ ! -f "$ORIGINAL_PACKAGE" ]]; then
  echo "File not found at path $1"
  exit 1
fi

cp $ORIGINAL_PACKAGE $WORKING_WHEEL

cd $TEMP_DIR

mkdir unzipped
unzip $WORKING_WHEEL -d unzipped

WHEEL_FILE=$(find unzipped -name "WHEEL")

grep -v '^Root-Is-Purelib' $WHEEL_FILE > $WHEEL_FILE.tmp
grep -v '^Tag' $WHEEL_FILE.tmp > $WHEEL_FILE
rm $WHEEL_FILE.tmp

echo "Root-Is-Purelib: true" >> $WHEEL_FILE
echo "Tag: py3-none-any" >> $WHEEL_FILE

cat $WHEEL_FILE

cd unzipped
zip -r ../universal_package.whl *
cd ..

cd $START_DIR

cp $TEMP_DIR/universal_package.whl $FINAL_PACKAGE

I've just started incorporating this into my own build pipelines to get around this bug.

[jrandall]

This should be fixed by python-poetry/poetry-core#227 (it has been working for us for months using a locally patched version). There seem to be a lot of open PRs for poetry and poetry-core, so I'm not sure how to get it reviewed.

[cjdsellers]

Just reporting this issue still occurs with Poetry 1.3.1.

Because our wheels package binaries compiled with Cython, we need to upload a wheel per Python version to PyPI.

After running poetry build --format wheel we need to fix the name of the wheel to respect the Python version it was created with. For example when building a wheel with Python 3.10 or 3.11, the wheel contains cp39 in the name (instead of cp310 or cp311 as applicable), and needs to be changed manually or with a script.

For further color, this occurs on both Linux and macOS, and pyenv exists for both environments I tested this on. It could be that Poetry uses the Python where it was installed for the file name, rather than the version of the virtualenv which is active?

[leotrs]

Same issue here, with poetry 1.3.1.

@cjdsellers care to share how you are changing the wheel names? Thanks!

[cjdsellers]

Same issue here, with poetry 1.3.1.

@cjdsellers care to share how you are changing the wheel names? Thanks!

Sure, simply just using mv 😄

[leotrs]

So no automated script? Do you also need to change the "Root-Is-Purelib: true" line like in this comment?

[cjdsellers]

So no automated script? Do you also need to change the "Root-Is-Purelib: true" line like in this comment?

I'm only modifying the file name, its also not a pure Python module like in the example from @tedivm

[leotrs]

Got it!

[tedivm]

My example is really if you want a universal wheel- if you are okay with your wheel being locked to a specific system than renaming it should be fine, but if you want it to actually be installable across multiple platforms (which for me was a requirement, as I had to support both ARM and AMD) then you need to update the wheel's internal files to reflect that.

[tedivm]

To be honest though I strongly recommend just not using poetry for building your stuff. It's too buggy and opinionated around local environments so it doesn't offer what's needed to true publishing. Utilizing the build project has been a much nicer experience.

[cjdsellers]

To be honest though I strongly recommend just not using poetry for building your stuff. It's too buggy and opinionated around local environments so it doesn't offer what's needed to true publishing. Utilizing the build project has been a much nicer experience.

Thanks for sharing!

[leotrs]

if you are okay with your wheel being locked to a specific system than renaming it should be fine, but if you want it to actually be installable across multiple platforms then you need to update the wheel's internal files to reflect that.

That's what I needed to hear. Thanks @tedivm !

[leotrs]

After many headaches, I have determined that changing the name of the wheel as recommended by previous comments is not enough in my case. I am trying to publish these wheel files to pypi and pypi will check the md5sum of each uploaded file. After manually changing the name, the md5 is of course still the same. I am moving away from poetry for the time being.

[cjdsellers]

After many headaches, I have determined that changing the name of the wheel as recommended by previous comments is not enough in my case. I am trying to publish these wheel files to pypi and pypi will check the md5sum of each uploaded file. After manually changing the name, the md5 is of course still the same. I am moving away from poetry for the time being.

If the hash is the same, it sounds like you may have a universal wheel though? i.e. you could potentially get away with a single wheel for all Python versions.

[leotrs]

I have a shared object file that needs to be compiled on different platforms.

And to clarify: the md5 mentioned above is not just the name. What I meant to say is that two files with different names have the same md5 because their contents are the same.

[cjdsellers]

Well I mean, this is definitely still a bug.

[leotrs]

Oh absolutely! Sorry for hijacking the thread 😅

[tedivm]

You didn't really hijack the thread, just pointed out another reason why poetry should not be used to build wheels that are meant to be published.

[finswimmer]

@tedivm: IMO your statement is to broad. If you mean "don't use Poetry to build wheels if you need a custom build script" I could agree, because build.py is still an unofficial feature. If you don't need a custom build script, Poetry works great for building wheels.

[domvwt]

Here's a temporary solution I also shared here:

#!/bin/bash

# Rename the wheel file to match the currently active python version
wheel_file=$(find dist/ -type f -name '*.whl')
echo "Renaming ${wheel_file}"
python_ver=$(python -c 'import platform; print("".join(platform.python_version_tuple()[:2]))')
echo "Using Python version: ${python_ver}"
new_wheel_file=$(echo ${wheel_file} | sed "s/-cp[0-9]*/-cp${python_ver}/g")
echo "New wheel file name: ${new_wheel_file}"
mv ${wheel_file} ${new_wheel_file}

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.