Strip Local version label (PEP 440) from Packages build from PackageInfo

[vedosis]

Resolves: #2613

• Added tests for changed code.
• Updated documentation for changed code.

Rebased into master i.l.o. 1.1 branch to avoid Python 2.7 (#4219)
RE: Documentation... I can't see anywhere the documentation would need to be updated. If it does, point me at it and I'll work it over.

New tests:

 poetry run pytest tests/
Test session starts (platform: linux, Python 3.8.5, pytest 6.2.4, pytest-sugar 0.9.4)
rootdir: /mnt/d/Source/poetry
plugins: cov-2.12.0, mock-3.6.1, sugar-0.9.4
collecting ...
 tests/test_factory.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                                         2% ▎
 tests/config/test_config.py ✓✓✓✓✓✓✓                                                                                                                                                                                                           3% ▍
 tests/console/test_application.py ✓✓✓✓                                                                                                                                                                                                        3% ▍
 tests/console/commands/test_about.py ✓                                                                                                                                                                                                        4% ▍
 tests/console/commands/test_add.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                              7% ▊
                                    ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                             11% █▎
                                    ✓✓✓                                                                                                                                                                                                       11% █▎
 tests/console/commands/test_cache.py ✓✓                                                                                                                                                                                                      12% █▎
 tests/console/commands/test_check.py ✓✓                                                                                                                                                                                                      12% █▎
 tests/console/commands/test_config.py ✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                             13% █▍
 tests/console/commands/test_export.py ✓✓✓✓✓✓                                                                                                                                                                                                 14% █▌
 tests/console/commands/test_init.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                      17% █▋
 tests/console/commands/test_lock.py ✓✓✓                                                                                                                                                                                                      17% █▊
 tests/console/commands/test_new.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                         19% █▉
 tests/console/commands/test_publish.py ✓✓✓✓✓                                                                                                                                                                                                 20% ██
 tests/console/commands/test_remove.py ✓                                                                                                                                                                                                      20% ██
 tests/console/commands/test_run.py ✓✓                                                                                                                                                                                                        20% ██
 tests/console/commands/test_search.py ✓                                                                                                                                                                                                      20% ██▏
 tests/console/commands/test_show.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                      23% ██▍
 tests/console/commands/test_version.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                             26% ██▋
 tests/console/commands/debug/test_resolve.py ✓✓✓                                                                                                                                                                                             27% ██▋
 tests/console/commands/env/test_info.py ✓✓                                                                                                                                                                                                   27% ██▋
 tests/console/commands/env/test_list.py ✓✓✓                                                                                                                                                                                                  27% ██▊
 tests/console/commands/env/test_remove.py ✓✓                                                                                                                                                                                                 27% ██▊
 tests/console/commands/env/test_use.py ✓✓✓                                                                                                                                                                                                   28% ██▊
 tests/console/commands/plugin/test_add.py ✓✓✓✓✓✓✓                                                                                                                                                                                            29% ██▉
 tests/console/commands/plugin/test_remove.py ✓✓                                                                                                                                                                                              29% ██▉
 tests/console/commands/plugin/test_show.py ✓✓✓                                                                                                                                                                                               29% ██▉
 tests/console/commands/self/test_update.py ✓✓                                                                                                                                                                                                30% ██▉
 tests/console/commands/source/test_add.py ✓✓✓✓✓✓                                                                                                                                                                                             30% ███▏
 tests/console/commands/source/test_remove.py ✓✓                                                                                                                                                                                              31% ███▏
 tests/console/commands/source/test_show.py ✓✓✓✓                                                                                                                                                                                              31% ███▎
 tests/inspection/test_info.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                          34% ███▌
 tests/installation/test_authenticator.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                   36% ███▋
 tests/installation/test_chef.py ✓✓✓                                                                                                                                                                                                          37% ███▋
 tests/installation/test_chooser.py ✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                              38% ███▊
 tests/installation/test_executor.py ✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                            40% ████
 tests/installation/test_installer.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓s✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                47% ████▊
 tests/installation/test_installer_old.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓s✓✓✓✓✓✓✓✓                                                                                                                                                  54% █████▍
 tests/installation/test_pip_installer.py ✓✓✓✓✓✓✓✓                                                                                                                                                                                            55% █████▌
 tests/masonry/builders/test_editable_builder.py ✓✓✓✓                                                                                                                                                                                         55% █████▌
 tests/mixology/solutions/providers/test_python_requirement_solution_provider.py ✓✓                                                                                                                                                           55% █████▋
 tests/mixology/solutions/solutions/test_python_requirement_solution.py ✓                                                                                                                                                                     56% █████▋
 tests/mixology/version_solver/test_backtracking.py ✓✓✓✓✓✓✓✓                                                                                                                                                                                  57% █████▋
 tests/mixology/version_solver/test_basic_graph.py ✓✓✓✓                                                                                                                                                                                       57% █████▊
 tests/mixology/version_solver/test_python_constraint.py ✓                                                                                                                                                                                    57% █████▊
 tests/mixology/version_solver/test_unsolvable.py ✓✓✓✓✓                                                                                                                                                                                       58% █████▊
 tests/mixology/version_solver/test_with_lock.py ✓✓✓✓✓                                                                                                                                                                                        59% █████▉
 tests/packages/test_locker.py ✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                                  60% ██████
 tests/plugins/test_plugin_manager.py ✓✓✓                                                                                                                                                                                                     61% ██████▏
 tests/publishing/test_publisher.py ✓✓✓✓✓✓✓✓                                                                                                                                                                                                  62% ██████▎
 tests/publishing/test_uploader.py ✓✓✓✓                                                                                                                                                                                                       62% ██████▎
 tests/puzzle/test_provider.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                          65% ██████▌
 tests/puzzle/test_solver.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓s✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                 78% ███████▊
 tests/repositories/test_installed_repository.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                               79% ███████▉
 tests/repositories/test_legacy_repository.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                       83% ████████▍
 tests/repositories/test_pool.py ✓✓✓✓✓✓✓                                                                                                                                                                                                      84% ████████▍
 tests/repositories/test_pypi_repository.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                   86% ████████▋
 tests/utils/test_env.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                            91% █████████▏
 tests/utils/test_env_site.py ✓✓                                                                                                                                                                                                              91% █████████▎
 tests/utils/test_exporter.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                95% █████████▌
 tests/utils/test_extras.py ✓✓✓✓✓✓✓                                                                                                                                                                                                           96% █████████▋
 tests/utils/test_helpers.py ✓✓✓                                                                                                                                                                                                              97% █████████▋
 tests/utils/test_password_manager.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                                                                                                                         99% █████████▉
 tests/utils/test_pip.py ✓✓                                                                                                                                                                                                                   99% █████████▉
 tests/utils/test_setup_reader.py ✓✓✓✓✓✓✓✓✓                                                                                                                                                                                                  100% ██████████
==============================
---- >8  ----  Warnings removed.  ---- 8< ----
==============================
Results (126.20s):
     763 passed
       3 skipped

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[vedosis]

The failures in CirrusCI look to be network driven. I don't think I changed anything there. Can I get someone to kick off the tests again to see if it's transitory?

[YoniChechik]

any news about this great fix? can help a lot when installing pytorch!

[vedosis]

@sdispater Maybe you can take a look at this PR to see? It looks like all the checks pass.

[kaustubhharapanahalli]

Any updates on this fix? @sdispater @finswimmer

[psinger]

This might fix pytorch issues, would be great to merge. @sdispater

[EgShes]

Any updates? Need to install pytorch too ...

[danmackinlay]

Installing packages like pytorch is not currently supported and there is no known timeline for support. FWIW this patch seemed to nearly get included in the last point release but then didn’t.

if you need this feature urgently you could consider forking the poetry project repo and merging in the supporting pull request; the project from which the pull request came is nearly what you want, (although some URLs might need updating to install correctly.) The commenters in this pull request and in #2613 might possibly collaborate on maintaining that fork?

[neersighted]

I'd like to see these two combined into a parameterized test, and then this should be good to go.

[vedosis]

<mickey-mouse-voice>I got three in one blow!</mickey-mouse-voice>

[neersighted]

Hate to do this to you @vedosis, but I think I might have overlooked some details in my earlier review. I'm wondering if we've considered the implications of a dependency requiring a +local_version, or the user requiring one.

I think the design might need to be rethought -- namely, the version parsing/comparison code should be the right target. poetry-core has a proper grammar for parsing versions, and modifications to the version comparison logic are likely to yield the best result.

Finally, I think that better testing is needed -- namely, we need to test what poetry does (or what is even expected) when nothing but +local_versions are available and the user requests none, requests one, and what happens when a dependency does the same.

Feel free to ask questions, work on this PR, or start a new one. I or @abn may give this a crack, but that is unlikely in the near future, so any interested parties are invited submit PRs along the lines of the above.

[vedosis]

Well... that's disappointing.

When I didn't get a response on the PR for a while, I moved on from poetry and resorted to a Frankensteinein Monster of shell scripts and pipenv. During my dive into the open-source abyss that is ML packages and +localVersions I discovered lots of people have strong opinions about what it should not be and like to tell other people they're wrong. Other people with opinions about what it could be are too afraid of voicing them lest there be some unintended consequence. PSF has asserted what they believe to be the intended state in PEP-440 (no local versions published) but the solution (PEP-459) is not an ML Model friendly solution because the installation providers (pip, pipenv, poetry) don't have enough standardized hooks into the underlying hardware (cpu vs gpu vs cuda lib stack) to even understand which version to pull, let alone the Herculean effort it would take to get even just pytorch to update their package metadata.

So... I guess what I'm trying to say is... that's disappointing. I'm going to close this PR. If anyone else wants to pick up the pytorch (hehe...) feel free to use the diff as a starter.

[neersighted]

Well... that's disappointing.

When I didn't get a response on the PR for a while, I moved on from poetry and resorted to a Frankensteinein Monster of shell scripts and pipenv. During my dive into the open-source abyss that is ML packages and +localVersions I discovered lots of people have strong opinions about what it should not be and like to tell other people they're wrong. Other people with opinions about what it could be are too afraid of voicing them lest there be some unintended consequence. PSF has asserted what they believe to be the intended state in PEP-440 (no local versions published) but the solution (PEP-459) is not an ML Model friendly solution because the installation providers (pip, pipenv, poetry) don't have enough standardized hooks into the underlying hardware (cpu vs gpu vs cuda lib stack) to even understand which version to pull, let alone the Herculean effort it would take to get even just pytorch to update their package metadata.

So... I guess what I'm trying to say is... that's disappointing. I'm going to close this PR. If anyone else wants to pick up the pytorch (hehe...) feel free to use the diff as a starter.

The real issue here is there is no support for the level of metadata pytorch needs -- namely, tags to differentiate ML hardware/APIs. The current solution with +local_versions allows for multiple binary packages to exist, but there's still no way for Poetry to do the right thing here.

I think if we can come up with an expected set of behaviors we can improve the situation (e.g. keep poetry from uninstalling versions that only differ +locally) -- but there is no perfect solution possible (yet).

[vikigenius]

The least we can do is not force update for eg: 1.10.0+cu113 -> 1.10.0 every time the dependency resolver runs.

Simply stripping it like this PR does should be good enough according to PEP-440.

As it stands it is simply impossible to use poetry if pytorch is installed in your environment.

[psinger]

Is there any hope for this to ever get merged / solved?

[danmackinlay]

I doubt this will be fixed any time soon. Most of the potential champions have gotten discouraged and moved on, and those who remain are mostly wishing for it rather than pushing it forwards in code and committee meetings. The best option if you want multi-architecture poetry is still maintaining your own fork with this patch. (which btw is not that hard, if you want need to track poetry mainline.)

I suspect that if a tech firm of some kind were invested in using poetry to manage their ML workflow this issue might get traction because developers could spend their work time on it.
If you work for such an organisation, perhaps you could make a business case to your manager that you could fix it. Otherwise, just use pip. It's adequate — which I think is part of the problem. Does poetry plausibly save enough developer time compared to the alternatives that it is worth trying to wade into this kind of mess?

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.