Associate dependency with source

[mbarkhau]

I have two entries in my pyproject.toml of the following form.

dependencies = [
    ...
    "internal-package~=1.0",
]

[[tool.pdm.source]]
url = "https://gitlab.company.com/api/v4/projects/309/packages/pypi/simple"
verify_ssl = true
name = "custom-packages-1"

This possibly causes a name collision issue if somebody uploads a package called internal-package to pypi: (will pdm use the package from pypi or from my custom source?
It also appears to slow down pdm, as I presume pdm is searching for requests, pandas etc. in my custom source instead of only on pypi.

Both of these issues could be solved if there were a way to associate a particular dependency with a particular source.

[ElijahSink]

The way I've solved this issue is setting the name field to pypi such that packages from my custom index will always take precedence over those on the official PyPI. However, this requires that your package index be configured to redirect to PyPI for packages that it doesn't have. Perhaps this would work for you.

[mbarkhau]

I could do that in principle yes, but not with my current setup, not with the way gitlab has implemented per-project packaging. I would need a proxy that implements this multi-project fallback.

[mbarkhau]

It appears that gitlab does have a plan for this: https://gitlab.com/groups/gitlab-org/-/epics/3612

Our current plan is to add support for virtual registries for PyPI, which would allow you to group multiple endpoints with a single URL.

[mbarkhau]

Getting this back on track (disregarding PyPI proxy ideas), is this something that pdm could solve on the client side?

[frostming]

pdm is complying with PEP 621 as the project metadata format. There is no room for such information to put in line with the dependency. Unless we use a table under [tool.pdm] to store that extra information but it seems a bit weird and not straightforward to me.

[mbarkhau]

Fair enough. Closing as wontfix.

[pawamoy]

Another mitigation for this dependency confusion issue is to follow some pattern for naming your internal packages, such as reducing the chances a public package uses the same name. Something like org-team-project-name. You'd have to keep this pattern and name private/hidden/secret of course (not public).

[ElijahSink]

Another question: does PDM resolve packages from multiple indexes in a deterministic way?

Perhaps you could list your private index under the "pypi" name, and then list the official PyPI as an extra index.

[frostming]

What about allowing users to specify the "order" of the sources to check? If so, what would the name be? naming is a headache..

[ElijahSink]

Aren't TOML tables ordered already? You could just take advantage of the order that they are listed in the table. One tricky part would be figuring out at which priority the implicit official pypi source would be positioned. I suppose it could be assumed that the official PyPI comes first unless a source is listed with the name pypi in some other position.

Some examples of what I mean:

Single additional source

[tool.pdm]

[[tool.pdm.source]]
url = "https://pypi.private-domain.org/simple"
verify_ssl = true
name = "custom-index-1"

Results in a priority:

1. Official PyPI (Uses -i flag with pip or just lets it use the default)
2. custom-index-1 (Uses --extra-index-url flag with pip)

Multiple additional sources, with the official PyPI being overridden

[tool.pdm]

[[tool.pdm.source]]
url = "https://pypi.other-private-domain.org/simple"
verify_ssl = true
name = "custom-index-2"

[[tool.pdm.source]]
url = "https://pypi.private-domain.org/simple"
verify_ssl = true
name = "custom-index-1"

[[tool.pdm.source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

Results in a priority:

1. custom-index-2 (uses -i flag with pip)
2. custom-index-1 (uses --extra-index-url with pip)
3. Official PyPI as listed in the table, (uses --extra-index-url with pip)

This of course relies on pip respecting the order of --extra-index-url arguments when resolving packages.

[frostming]

@ElijahSink This is how it is working currently, but that the package version in the private index must be higher than PyPI to get chosen.

[ElijahSink]

the package version in the private index must be higher than PyPI to get chosen.

@frostming is that an implementation detail of pip or something that PDM can control?

[frostming]

@ElijahSink Unfortunately the former, pip fetches candidates from all indexes and sorts them together to get the best match. We already have a plan to implement our own package finder to replace pip's. So let's leave it for this release and do improvements in the future.

[ElijahSink]

@frostming Where is the right place to discuss the implementation of the new package finder?

[frostming]

It is just an idea in my mind, no issue or discussion to track.

[frostming]

Closed by #1155, please test it on 2.0.0a1