Poetry created wheel puts includes into root of site-packages folder & is missing included folders.

[MarximusMaximus]

• Poetry version: Poetry (version 1.2.2)
• Python version: Python: 3.10.6
• OS version and name: macOS 12.6
• pyproject.toml: https://gist.github.com/MarximusMaximus/19b84135b3da1f1670f50e06bc364d00

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

When installing a wheel created by poetry and having specified include= in pyproject.toml, the included files get extracted into the root of the site-packages folder.

Steps:

• add include= to pyproject.toml (see example)
• poetry build
• pip install
• ls -l

Alternative Steps for Verification:

• add include= to pyproject.toml (see example)
• poetry build
• cd
• mkdir sdist
• tar -xzf <built .tar.gz> -C sdist
• mkdir wheel
• tar -xzf <build .whl> -C wheel
• ls -l sdist
• ls -l wheel

Results:

• note that the files from the includes list are now in the site-packages folder/root extracted folder, not inside any of the built package folders
• additionally, included folders are missing from the wheel

Expected:

• the files from the include list would not be in site-packages/root extracted folder itself, but in one of the built package's folders [as occurs with the sdist built version]

sdist result:

./
../
pytest_shell_script_test_harness-0.1.0.dev0/
    CHANGELOG.rst       <---
    LICENSE             <---
    PKG-INFO
    README.md
    pyproject.toml      <---
    setup.py
    src/
    tests/              <---

wheel result:

./
../
CHANGELOG.rst           <---
LICENSE                 <---
pyproject.toml          <---
pytest_shell_script_test_harness/
    __impl.py
    __init__.py
    coverage_plugin.py
    py.typed
    resources/
pytest_shell_script_test_harness-0.1.0.dev0.dist-info/
    LICENSE
    METADATA
    RECORD
    WHEEL
    entry_points.txt

NOTE: 'tests' folder missing from wheel

[dimbleby]

including files that you have told poetry to include is not a bug. The fix is simply: don't do that.

not including tests in wheels is normal.

please close.

[MarximusMaximus]

They are being included in the WRONG PATH in the wheel. That is the bug.

If I specify to include the tests folder in the wheel, it should get included. Because I asked it to. But it does not. That is also what I'd call a bug -- it doesn't do what I asked.

[dimbleby]

CHANGELOG etc are not being put in the wrong path: that's how you've laid out your project and that's what you've told poetry to include. Neither poetry nor I can guess what else you might have intended.

(Probably you ought to include these files only in the sdist, which would make sense.)

As for the tests: I now see you have tried to include a whole directory, but that's not how that works. You should include tests/*. #2685 already exists asking to be able to include directories, we don't need this to duplicate that.

Edit: or, judging by poetry's own pyproject.toml, this would also work:

include = [{ path = "tests", format = "sdist" }]

Please close.

[MarximusMaximus]

CHANGELOG etc are not being put in the wrong path: that's how you've laid out your project and that's what you've told poetry to include. Neither poetry nor I can guess what else you might have intended.

Why would they be put in a different path in the wheel compared to the sdist then? They are being put into the root of site-packages when included in the wheel, which is generally a bad thing?

Edit to clarify: when part of the sdist, they get put inside the pytest_shell_script_test_harness/ folder, not the root of site-packages.

As for the tests: I now see you have tried to include a whole directory, but that's not how that works. You should include tests/*. #2685 already exists asking to be able to include directories, we don't need this to duplicate that.

Acknowledged.

[dimbleby]

when part of the sdist, they get put inside the pytest_shell_script_test_harness/ folder

that is the equivalent top-level part of the project tree in the sdist: adjacent to /src and /tests. Again: that is as you laid out your project.

Perhaps you are misreading the wheel. The pytest_shell_script_test_harness directory there is taken from the src directory, of course CHANGELOG etc don't go in there.

[RobbeSneyders]

I originally opened #8994, but that seemed to be a duplicate of this issue, so I will repeat the relevant info here.

• Poetry version: 1.7.1
• Python version: 3.10.12
• OS version and name: Ubuntu 22.04
• pyproject.toml: https://github.com/ml6team/fondant/blob/4a186967e860719e0385f0e1f5010814d8792e49/pyproject.toml

When building our package, we want to include only yaml files and ignore all other files from a certain subdirectory. The relevant part of our file tree looks like this:

├── src
|   ├── fondant
|   |   ├── components
|   |   |   ├── component_1
|   |   |   |  ├── fondant_component.yaml
|   |   |   |  └── bunch of other files and dirs
|   |   |   └── component_2
|   |   |   |  ├── fondant_component.yaml
|   |   |   |  └── bunch of other files and dirs
|   |   ├── some other directories
|   |   └── some python files
└── pyproject.toml

See our full file tree in our repo, the relevant directories and files are called the same.

We only want to include the fondant_component.yaml file from each component in src/fondant/components and ignore all other files.

I tried to do this with the following include / exclude in our pyproject.toml:

include = ["src/fondant/components/**/*.yaml"]
exclude = ["src/fondant/components"]

So excluding the full directory, and including the specific files we do want to package.

However this leads to inconsistent results in the sdist and wheel outputs:

sdist

src
├── fondant
    ├── components
    |   ├── component_1
    |   |  ├── fondant_component.yaml
    |   └── component_2
    |      ├── fondant_component.yaml
    ├── some other directories
    └── some python files

This is the result we want to achieve.

wheel

├── fondant
|   ├── some other directories
|   └── some python files
└──src
   └── fondant
       └── components
           ├── component_1
           |  ├── fondant_component.yaml
           └── component_2
              └── fondant_component.yaml

As a reproducible example, you can run poetry build from the root of our repo.

I've tried different things to achieve this result:

• Starting from the package as root as recommended in Include exclude globs with whitelist pattern #1336
• Using an exclude pattern for yaml in the exclude glob:

  include = []
  exclude = ["src/fondant/components/**/*[!.yaml]"]

but none of them worked.

Poetry's behavior is currently inconsistent:

• for include between sdist and wheel
  The include path is added as a different path in sdist and wheel. This leads to a working installation with sdist and a broken installation with wheel.
• for wheel between include and exclude
  I define both my include and exclude paths with src as the root. For exclude, the fact that the fondant package is located in the src directory is correctly taken into account, and fondant/components is excluded. For include, this is not taken into account, and src/fondant/components is included instead.

[dimbleby]

I think that you are muddying the waters: what is described by this issue report is just not a bug: this issue really should have been closed

what you have found is at least different, you have extra complications with src layout and with "exclude", it's not clear to me whether it's a bug or not

IMO the right admin here is to track your thing in the issue that you have just closed - where per my last I invite you to look at the code and figure out what is going on / contribute a fix.

[RobbeSneyders]

I do think my issue is a different manifestation of the same underlying issue, but I will reopen it. I would appreciate if you could confirm that it is a bug before I dive into an unknown code base. I have provided a detailed description with only the relevant parts of the issue, and you can pull and build our public code base with two commands to reproduce it.

[MarximusMaximus]

this issue really should have been closed

Apologies for that.
Related questions to the OG issue:
Is there a way to specify a target path for a file to end up in during include?
If I use a symlink in the target location and include the symlink instead of the real file, will the include have the real file or just a dangling symlink?

If it's not possible to do either of those, I would request the ability to alter the target location of an included file.

TY

[radoering]

Is there a way to specify a target path for a file to end up in during include?

Seems like it is (will be in 1.8) only possible for package includes but not for general includes.

[MarximusMaximus]

Doh. I'd need it for general includes.

[bassdr]

I need this too. This is very annoying and yes, either I dont understand why you are saying this is not a bug... I end up with files clashing in my site-packages because one package wants to include a file (LICENSE) at the root of site-packages. This is not allowed by my package manager to avoid obvious file clashes between packages.

I'm not the author of this package, only tring to create a package for the package manager I'm using, so I'm ok to say this is not a bug if there is a simple way to patch the broken pyproject.toml. I don't think I can drop the LICENSE file legally. if the owner wants it in the package it should be somewhere in the package...

FTR, the package manager I'm writing a package for is gentoo's portage, and the problematic python project is pwmled, a dependency of homeassistant.

[radoering]

@bassdr You do not need it for LICENSE files. Of course, LICENSE files should be put into the sdist and the wheel. However, in the wheel, they must not be put top-level but in the .dist-info directory. Poetry does this automatically for you. Here you can see which files are handled automatically.

the problematic python project is pwmled

You can tell them that these lines should be removed. The LICENSE file will still be included top-level in the sdist and in the .dist-info folder in the wheel. Just try it.