Bad installation location for metadata files

[sduenas]

Metadata files such as AUTHORS, NEWS, etc, aren't correctly installed when the package is installed using pip.

This is the content of the RECORD file, as you can see, AUTHORS and so on are installed in the root directory (normally, lib/python<version>/site-packages.

AUTHORS,sha256=nVlSiMmkhiF6Tz0u1pglGeVH_pNFRatM-qRwIy1NqUk,218
LICENSE,sha256=jOtLnuWt7d5Hsx6XXB2QxzrSe2sWWh3NgMfFRetluQM,35147
NEWS,sha256=xxt-0J75kqWvtnY5kZsT05rFeKLaeNAzsaJup-FiizY,503
README.md,sha256=423qaRZHZad2ewatXkJYEB5J82edWrjO4vy4gJyjfYI,842
grimoirelab_toolkit-0.2.0.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
grimoirelab_toolkit-0.2.0.dist-info/LICENSE,sha256=jOtLnuWt7d5Hsx6XXB2QxzrSe2sWWh3NgMfFRetluQM,35147
grimoirelab_toolkit-0.2.0.dist-info/METADATA,sha256=7fQ7WqC8VUnmwU6fo6UJw7Ajdq6e8BRUNeGJSfJtudI,1751
grimoirelab_toolkit-0.2.0.dist-info/RECORD,,
grimoirelab_toolkit-0.2.0.dist-info/WHEEL,sha256=V7iVckP-GYreevsTDnv1eAinQt_aArwnAxmnP0gygBY,83
grimoirelab_toolkit/__init__.py,sha256=NS1vGJp1OVlLNXBuUCVJYsoCsXyHJUiSB7dOg23eGGE,1011
grimoirelab_toolkit/__pycache__/__init__.cpython-38.pyc,,
grimoirelab_toolkit/__pycache__/_version.cpython-38.pyc,,
grimoirelab_toolkit/__pycache__/datetime.cpython-38.pyc,,
grimoirelab_toolkit/__pycache__/introspect.cpython-38.pyc,,
grimoirelab_toolkit/__pycache__/uris.cpython-38.pyc,,
grimoirelab_toolkit/_version.py,sha256=5_9aOJ6JVM57sF2Am2VNkKo3OXP5y-uKXinXcsLC1Fw,97
grimoirelab_toolkit/datetime.py,sha256=8Tp9mCtyXvpNmUke0yG6GiB4iFEcEr3J7vSKrQaJpOo,5606
grimoirelab_toolkit/introspect.py,sha256=gK0T4tUMB9TezN8Bg5SVE6z8_NUFhV3iixp_ZeDPFpE,3846
grimoirelab_toolkit/uris.py,sha256=KBmHY8G8vaOleSzYc2pHXcyWEUSlIUpv4Qp6mymSrlw,1383

You can reproduce this error creating a new environment and instlalling the package:

$ python3 -m venv /tmp/myvenv
$ source /tmp/myvenv/bin/activate/
$ pip install grimoirelab-toolkit
$ ls /tmp/myvenv/lib/python3.8/site-packages/NEWS
/tmp/myvenv/lib/python3.8/site-packages/NEWS

[vchrombie]

Hi @sduenas, thanks for opening the issue.

$ python3 -m venv /tmp/myvenv 
$ source /tmp/myvenv/bin/activate/ 
$ pip install grimoirelab-toolkit 
$ ls /tmp/myvenv/lib/python3.8/site-packages/NEWS 
/tmp/myvenv/lib/python3.8/site-packages/NEWS

I was able to reproduce the error.

I tried building the packages and experimenting with them. It looks like the error is because of the (faulty?) wheel packages.

~ $ git clone https://github.com/chaoss/grimoirelab-toolkit
~ $ cd grimoirelab-toolkit
~/grimoirelab-toolkit $ poetry install
~/grimoirelab-toolkit $ poetry shell

(.venv) $ poetry build
Building grimoirelab-toolkit (0.2.0)
  - Building sdist
  - Built grimoirelab-toolkit-0.2.0.tar.gz
  - Building wheel
  - Built grimoirelab_toolkit-0.2.0-py3-none-any.whl

(.venv) $ python -m tarfile -l dist/grimoirelab-toolkit-0.2.0.tar.gz
grimoirelab-toolkit-0.2.0/AUTHORS 
grimoirelab-toolkit-0.2.0/LICENSE 
grimoirelab-toolkit-0.2.0/NEWS 
grimoirelab-toolkit-0.2.0/README.md 
grimoirelab-toolkit-0.2.0/grimoirelab_toolkit/__init__.py 
grimoirelab-toolkit-0.2.0/grimoirelab_toolkit/_version.py 
grimoirelab-toolkit-0.2.0/grimoirelab_toolkit/datetime.py 
grimoirelab-toolkit-0.2.0/grimoirelab_toolkit/introspect.py 
grimoirelab-toolkit-0.2.0/grimoirelab_toolkit/uris.py 
grimoirelab-toolkit-0.2.0/pyproject.toml 
grimoirelab-toolkit-0.2.0/tests/run_tests.py 
grimoirelab-toolkit-0.2.0/tests/test_datetime.py 
grimoirelab-toolkit-0.2.0/tests/test_introspect.py 
grimoirelab-toolkit-0.2.0/tests/test_uris.py 
grimoirelab-toolkit-0.2.0/setup.py 
grimoirelab-toolkit-0.2.0/PKG-INFO 

(.venv) $ python -m zipfile -l dist/grimoirelab_toolkit-0.2.0-py3-none-any.whl
File Name                                             Modified             Size
AUTHORS                                        1980-01-01 00:00:00          218
LICENSE                                        1980-01-01 00:00:00        35147
NEWS                                           1980-01-01 00:00:00          503
README.md                                      1980-01-01 00:00:00          842
grimoirelab_toolkit/__init__.py                1980-01-01 00:00:00         1011
grimoirelab_toolkit/_version.py                1980-01-01 00:00:00           97
grimoirelab_toolkit/datetime.py                1980-01-01 00:00:00         5606
grimoirelab_toolkit/introspect.py              1980-01-01 00:00:00         3846
grimoirelab_toolkit/uris.py                    1980-01-01 00:00:00         1383
grimoirelab_toolkit-0.2.0.dist-info/LICENSE    1980-01-01 00:00:00        35147
grimoirelab_toolkit-0.2.0.dist-info/WHEEL      2016-01-01 00:00:00           83
grimoirelab_toolkit-0.2.0.dist-info/METADATA   2016-01-01 00:00:00         1751
grimoirelab_toolkit-0.2.0.dist-info/RECORD     2016-01-01 00:00:00         1032

The whl package is inflated into the /tmp/myvenv/lib/python3.8/site-packages/ folder directly when installed using pip. So, the files AUTHORS, NEWS, LICENSE (duplicate tho), and README.md are seen in the root directory as you mentioned earlier.

This is the content of the RECORD file, as you can see, AUTHORS and so on are installed in the root directory (normally, lib/python/site-packages.

.
|____grimoirelab_toolkit
| |____uris.py
| |_____version.py
| |____datetime.py
| |______init__.py
| |____introspect.py
|____grimoirelab_toolkit-0.2.0-py3-none-any.whl
|____LICENSE
| |____WHEEL
| |____METADATA
| |____LICENSE
| |____RECORD
|____AUTHORS
|____NEWS
|____README.md

I guess they have to be present in the grimoirelab_toolkit-0.2.0.dist-info folfer. Do you think this is the expected format?

.
|____grimoirelab_toolkit
| |____uris.py
| |_____version.py
| |____datetime.py
| |______init__.py
| |____introspect.py
|____grimoirelab_toolkit-0.2.0.dist-info
| |____WHEEL
| |____METADATA
| |____LICENSE
| |____RECORD
| |____AUTHORS
| |____NEWS
| |____README.md

[sduenas]

@vchrombie correct. I think they should be under grimoirelab_toolkit-0.2.0.dist-info if possible. What it's sure they don't go under site-packages.

[vchrombie]

I think they should be under grimoirelab_toolkit-0.2.0.dist-info if possible.

Yup, makes sense to me.

What it's sure they don't go under site-packages.

I agree with this.

Not very sure, but looks like the issue is from the poetry side. I will open an issue in poetry repository to see if anyone experiences the same issue or if there is any workaround.

[sduenas]

I hope there's a workaround to fix this problem because if we need to wait for poetry, we might not have it in a stable release for the next 2 years.

[vchrombie]

Update from python-poetry/poetry#4494 (comment)

it's not that easy as I first thought. We cannot put any file included from the root folder into dist-info, because in case of "module only" packages, we exactly want the current behavior.

storing the files in the root directory is something that is needed by the poetry

Further investigation shows, that poetry already include files from the root folder, that matches the pattern COPYING* and LICENSE*, into the dist-info folder of the wheel. So just remove those entries from the includelist.

yes, I can confirm this. The LICENSE file is added by default for both sdist and wheel

Including license information into the wheel makes absolutely sense for legal reasons. Other files like README or NEWS are more for sdist's. That's why I think there is no strong reason for an option to include in arbitrary number of files to the dist-info folder of a wheel.

if we agree to this point, we can change the pyproject.toml configurations as per below

diff --git a/pyproject.toml b/pyproject.toml
index 1bb0b22..b91cdaa 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -23,10 +23,9 @@ packages = [
 ]
 
 include = [
-    "AUTHORS",
-    "README.md",
-    "LICENSE",
-    "NEWS"
+    { path = "AUTHORS", format = "sdist" },
+    { path = "NEWS", format = "sdist" },
+    { path = "README.md", format = "sdist" },
 ]
 
 classifiers = [

This would give us the below results for the packaging.

$ poetry build
Building grimoirelab-toolkit (0.2.0)
  - Building sdist
  - Built grimoirelab-toolkit-0.2.0.tar.gz
  - Building wheel
  - Built grimoirelab_toolkit-0.2.0-py3-none-any.whl

$ python -m tarfile -l dist/grimoirelab-toolkit-0.2.0.tar.gz
grimoirelab-toolkit-0.2.0/AUTHORS 
grimoirelab-toolkit-0.2.0/LICENSE 
grimoirelab-toolkit-0.2.0/NEWS 
grimoirelab-toolkit-0.2.0/README.md 
grimoirelab-toolkit-0.2.0/grimoirelab_toolkit/__init__.py 
grimoirelab-toolkit-0.2.0/grimoirelab_toolkit/_version.py 
grimoirelab-toolkit-0.2.0/grimoirelab_toolkit/datetime.py 
grimoirelab-toolkit-0.2.0/grimoirelab_toolkit/introspect.py 
grimoirelab-toolkit-0.2.0/grimoirelab_toolkit/uris.py 
grimoirelab-toolkit-0.2.0/pyproject.toml 
grimoirelab-toolkit-0.2.0/tests/run_tests.py 
grimoirelab-toolkit-0.2.0/tests/test_datetime.py 
grimoirelab-toolkit-0.2.0/tests/test_introspect.py 
grimoirelab-toolkit-0.2.0/tests/test_uris.py 
grimoirelab-toolkit-0.2.0/setup.py 
grimoirelab-toolkit-0.2.0/PKG-INFO 

$ python -m zipfile -l dist/grimoirelab_toolkit-0.2.0-py3-none-any.whl
File Name                                             Modified             Size
grimoirelab_toolkit/__init__.py                1980-01-01 00:00:00         1011
grimoirelab_toolkit/_version.py                1980-01-01 00:00:00           97
grimoirelab_toolkit/datetime.py                1980-01-01 00:00:00         5606
grimoirelab_toolkit/introspect.py              1980-01-01 00:00:00         3846
grimoirelab_toolkit/uris.py                    1980-01-01 00:00:00         1383
grimoirelab_toolkit-0.2.0.dist-info/LICENSE    1980-01-01 00:00:00        35147
grimoirelab_toolkit-0.2.0.dist-info/WHEEL      2016-01-01 00:00:00           83
grimoirelab_toolkit-0.2.0.dist-info/METADATA   2016-01-01 00:00:00         1751
grimoirelab_toolkit-0.2.0.dist-info/RECORD     2016-01-01 00:00:00          779

$ cat dist/grimoirelab_toolkit-0.2.0.dist-info/RECORD
grimoirelab_toolkit/__init__.py,sha256=NS1vGJp1OVlLNXBuUCVJYsoCsXyHJUiSB7dOg23eGGE,1011
grimoirelab_toolkit/_version.py,sha256=5_9aOJ6JVM57sF2Am2VNkKo3OXP5y-uKXinXcsLC1Fw,97
grimoirelab_toolkit/datetime.py,sha256=8Tp9mCtyXvpNmUke0yG6GiB4iFEcEr3J7vSKrQaJpOo,5606
grimoirelab_toolkit/introspect.py,sha256=gK0T4tUMB9TezN8Bg5SVE6z8_NUFhV3iixp_ZeDPFpE,3846
grimoirelab_toolkit/uris.py,sha256=KBmHY8G8vaOleSzYc2pHXcyWEUSlIUpv4Qp6mymSrlw,1383
grimoirelab_toolkit-0.2.0.dist-info/LICENSE,sha256=jOtLnuWt7d5Hsx6XXB2QxzrSe2sWWh3NgMfFRetluQM,35147
grimoirelab_toolkit-0.2.0.dist-info/WHEEL,sha256=xSvaL1DM8LOHfdyo0cCcwjZu1tC6CnCsRGWUgazvlbM,83
grimoirelab_toolkit-0.2.0.dist-info/METADATA,sha256=7fQ7WqC8VUnmwU6fo6UJw7Ajdq6e8BRUNeGJSfJtudI,1751
grimoirelab_toolkit-0.2.0.dist-info/RECORD,,

The METADATA file contains the README.md instructions.

Please let me know what do you think. I can open the PR if needed.