Home

A Roadmap towards better Python Packaging

Or in other words, a top down view of big problems to solve in Python Packaging.

Disclaimer: currently an overview by Matthew Iversen, not at all the canonical opinion of the Python Packaging Authority.

Metadata

Versions:

• 1.0 - key/value
• 1.1 - key/value
• 1.2 - key/value - initial support distribution depedency
• 2.0 - json

Use:

• Setuptools puts METADATA 1.0 in *.egg-info/METADATA's
• Wheel 1.0 puts METADATA "2.0" in *.dist-info/METADATA
• pydist.json supposedly holds METADATA 2.0

Problems:

• Change wheel to indicate it's saving METADATA 1.2?
• Update setuptools to output METADATA 1.2 information
• Work towards allowing more declarative specification of metadata in packages, than that currently encourages by setuptools
• Figure out best practices for platform and ABI compatibility, requires a lot of communication, but currently going ok

Dependency Resolution

Problems:

• Pip has no ordering of indexes from which to pull packages
• No resolution of conflicting dependencies
• No central database of installations (currently separate .{egg,dist}-info folders)
• Packages must be fully downloaded to discover their dependencies

Solutions:

• Introduce a proper dependency resolution solver to pip
• Ingestion of package metadata for dependency information could be separated from download package contents, to make installs faster and more customizable.
• Allow for more descriptive version and dependency information in metadata (python versions, implementations, platforms, abis, etc).
• A dependency solver needs a reliable database of of what is already installed and all dependency metadata relating to that. Currently metadata 1.2 is minimum viable for this, but 2.0 would be preferred for platform/abi/etc dependencies

Prior art

Packages, Installation and Formats

Wheel:

For future Wheel-Versions

• Add a controlled, sanctioned way of making zip-importable wheels?
• • • • • • • executable wheels? (python mywheel.whl or python -m mywheel.whl?)
• Fix and version metadata inclusion
• wheel filenames are currently used to tell a lot of compatibility information - is this appropriate? When you start to practically factor in platforms and ABIs, people can get huge filenames. Where / when should the line be drawn? Could this be moved away from?

Platforming and ABIs

• Discussion on OS X ABI compatibility between its pythons

Setuptools

• Update output information to metadata 1.2, to work towards viable dependency information in local package database

Logistics and Documentation

In progress

• virtualenv docs update
• Completing PPUG - some of our best work here!

Organisation of Mailing Lists & Communication Channels

• Freenode irc: #pypa, #pip, #pypa-dev, #distutils
• Mailing-lists: [email protected], pypa-dev, python-virtualenv, others?

These could be restructured into two central channels per communication medium - one for users to discuss problems and questions (e.g "pypa-users" or "pypa"), and one for developers to discuss work on projects (e.g "pypa-dev"). If so, such reorganisation should be announced to people!

Furthering Communication

This needs to be improved, a lot. See

• setuptools Features
• Lack of Communication
• Lack of good deprecation policies

I'm sure there are quite a few other examples I've missed that all could be addressed systematically.