Manage projects with namespaces

[4383]

Hey!

Overview

This is a feature proposal. Now when a project is already register on pypi it's not possible to users to test a fork of any projects with the same name when it's already exist, manage projects by namespace increase possiblities for the python community.

With this feature we can introduce trusted packages by allow install/search without namespace and add namespaces on untrusted packages like docker behavior (docker pull nginx or docker pull 4383/nginx).

On docker when the package is trusted (docker trusted image mean maintained by docker itself), namespace does not exist, and when a package is maintain by a third user namespace appear into the name.

I don't want delegate official projects maintainance to the pypa team but we can introduce a vote system for the community and remove namespace when project obtain a certain number of votes from the community (users).

I'm sure to be in the right place for propose this feature, twine, pip etc... must be upgraded also for work with it.

Features

• Allow users to vote for trust project and allow download (install, search, etc...) without prefix with user namespace
• Allow users to upload on pypi project with a name who already exist on pypi but prefixed by user namespace.

Benefits

• Improve project trust
• Improve package trusting and discrease risk that users deal with a miscellaneous package come from a typo squatting example 1, pypa github discussion
• Allow users to provide forked version of an official project
• Allow users to test that packaging work fine on pypi

Examples

With pip:

$ pip install Django # trusted package
$ pip install 4383/Django # untrusted package

Url transposition:

• https://pypi.org/project/Django/
• https://pypi.org/project/4383/Django/

[brainwane]

@4383 Thank you for this feature proposal! I'm the project manager for the current push to deploy Warehouse and, since this is a new feature idea, I've marked it for a future milestone, since it isn't a task we absolutely need to do in order to get Warehouse deployed. Thanks again!

[4383]

@brainwane how I can help you for introduce this feature?

[brainwane]

@4383 I'm so sorry for the wait! @di do you have any thoughts on what next steps would be?

[di]

This will need discussion, likely on distutils-sig. Since a package name like 4383/Django is not currently a valid package name, this would possibly require a PEP, as well as significant changes to tooling in the ecosystem (pip, twine, etc).

[4383]

Hi @brainwane @di!

I can describe the attended behavior in PEP for more examples and to standardize this feature.
My help can start by writing this description.

Thanks for replies!

[di]

To be clear: a discussion on distutils-sig should probably be the first step.

[4383]

@di ok

[4383]

Discussion on distutils-sig was now opened
https://groups.google.com/forum/#!topic/distutils-sig/gFusrdTuWek

And message was sent on the disutils-sig mailing list

[brainwane]

I think that might be the wrong list. Please post to https://mail.python.org/mailman/listinfo/distutils-sig . Thank you and sorry for the mixup.

[4383]

I had already sent an email to this mailing list: [email protected]
Can you confirm it's right?

[brainwane]

Thanks! I see it at https://mail.python.org/mm3/archives/list/[email protected]/thread/NFLCHKYGPEBAR5I5FMMXMC2JKMLYJQ7G/ in the archives of the distutils-sig mailing list. :)

If you will be at PyCon North America or EuroPython in Edinburgh this year, please consider talking with us in person there at our sprints.

[ewdurbin]

cc @pganssle

[4383]

Hello everybody!
No response from the distutils team about this feature proposal.
I've started to write a PEP for define more properly this feature and to start thinking more deeply and share with the community.
Currently it's a draft and I ask to me a lot of questions about the voting system or a method like this.

python/peps#686

[di]

Relevant discussion thread here: https://discuss.python.org/t/namespace-support-in-pypi/1609

[4383]

@di oh thanks! Nice to see similar discussions, I'll take a look

[brainwane]

There's been substantial discussion of this suggestion on the Discourse forum and I suggest interested people take a look.

[cowlinator]

@brainwane

As @di and @DataNerdery mentioned, a PyPI user named "RemindSupplyChainRisks" uploaded 3591 packages to PyPI that point to a malicious URL. A lot of the names are things like "cupy-cuda112" or other common typos.

See pypi/support#923
and see also https://twitter.com/DataNerdery/status/1366263351685226499

This feature would allow a user to reserve all similar names ("typo names") of a package, preventing this situation.

I think this feature should be considered a security feature, and thus prioritized.

[elsiehupp]

I was initially going to post the following comment on #201, which overlaps somewhat, and this was discussed to some extent on the above-linked Discourse thread, as well as in another Discourse thread, though it doesn’t not seem to be covered by PEP 541.

While doing so might require a specific implementation in Python itself, perhaps one of the following could work:

• Allow PyPI users to authenticate via GitHub and/or Gitlab (potentially alongside a manual approval process for adding other repository platforms) to register a name prefix, so that something like https://pypi.org/teams/<team_name>/<package_name>/ and/or https://pypi.org/users/<user_name>/<package_name>/ along with pip install <team_name>/<package_name> and/or pip install <user_name>/<package_name> could be used.
• Allow for the use of “reverse domain name” prefixes in package names, like, e.g., @flathub does, potentially with some degree of verification to prevent abuse. (Flathub manually approves additions to their index, which mitigates the issue of illegitimate prefix-squatting.) In the case of Flathub, GitHub-based packages without their own websites use the prefix io.github.*, which redirects to GitHub Pages, leaving com.github.* de facto reserved for the GitHub organization itself.

I say that this might require a specific implementation in Python itself because I don’t know how Python import … statements would support namespace prefixes. I know Java sometimes uses nested folders to emulate the “reverse domain name" hierarchy in a given codebase, so it’s possible this could be implemented in pip without requiring changes to the Python library itself.

FWIW, supporting (optional) domain-style prefixes on PyPI and in pip could also be useful if, for example, a particular developer needs to import both the canonical, published version of a module and an as-yet unmerged fork into the same Python environment, which could otherwise be even more of a mess.

The main benefit to using “reverse domain name” prefixes is that it eliminates the conflict between multiple username-authentication platforms (e.g. GitHub vs Gitlab, etc.) in favor of delegating name ownership to the existing ICANN registrar/DNS ecosystem. In other words, it kicks responsibility down the road to someone else. Another potential benefit to using “reverse domain name” style prefixes is that it could mitigate the potential issue of team names conflicting with package names, though I’m sure there are other factors to consider. (Yes, I know it’s possible to set up third-party or private repositories for use with pip, but that’s an unrealistically imposing threshold for a lot of developers.)

On the flip side, authenticating GitHub and/or Gitlab usernames benefits from the fact that both platforms already have authentication APIs, so automatic namespace authentication (should that be desirable), wouldn’t require someone to, say, implement a <link/> or <meta/> header on a personal website in order to have a protected namespace. However, with “reverse domain name” namespaces, GitHub and/or Gitlab authentication could just be used for io.github.* and io.gitlab.* domain prefixes, so a hybrid approach would be possible.

Finally, as a change to pip itself, namespace collision (and typosquatting) could be mitigated by automating some sort of two-step installation process, where typos or unqualified/unprefixed package names could return a prompt with a list of options. This is the approach taken with flatpak install where, e.g., you get the following command flow:

$ sudo flatpak install gnome
[sudo] password for elsiehupp: 
Looking for matches…
Found similar ref(s) for ‘gnome’ in remote ‘flathub’ (system).
Use this remote? [Y/n]: y
Similar refs found for ‘gnome’ in remote ‘flathub’ (system):

   1) runtime/org.gnome.Sdk.Docs/x86_64/3.32
   2) runtime/org.gnome.Sdk.Docs/x86_64/3.34
   3) runtime/org.gnome.Sdk.Docs/x86_64/3.36

   […]

  174) runtime/org.gnome.Sdk.Docs/x86_64/3.28
  175) runtime/org.gnome.Platform.Compat.i386/x86_64/40
  176) runtime/org.gnome.Sdk.Docs/x86_64/3.30

Which do you want to use (0 to abort)? [0-176]: 

This command flow has the added benefit of also being a search interface, so the user would have less of a need to visit the repository website to resolve the difference between similar-sounding package names. (Implementing more of a search interface into pip would be a separate issue, though it’s one I’ve really not looked into.)

[AaronFriel]

Weighing in on this, we at Pulumi were unable to publish a package today because pulumi-local is occupied, this leads to a bit of confusion as we were able to publish, e.g., @pulumi/local on NPM and Pulumi.Local on NuGet.

Namespaces would assist us in growing our ecosystem without having to worry about squatting. (This, to be clear, is not an instance of squatting, but an unfortunate collision.)