REP 151 Python 3 rewrite

rep-0151.rst

@@ -0,0 +1,260 @@
+REP: 151
+Title: Python 3 Support
+Author: Dirk Thomas, Shane Loretz
+Status: Draft
+Type: Standards Track
+Content-Type: text/x-rst
+Created: 05-Aug-2019
+
+Outline
+=======
+
+#. Abstract_
+#. Motivation_
+#. Context_
+#. Proposal_
+#. References_
+#. Copyright_
+
+Abstract
+========
+
+This REP describes the path for switching from Python 2 to Python 3 in ROS 1.
+
+Motivation
+==========
+
+As described in the Python release schedule [1]_ Python 2.7 will only receive
+bugfix releases until 2020.
+As a consequence Ubuntu is trying to demote Python 2.7 into the `universe`
+repository [2]_.
+While it is unlikely that Python 2.7 will be removed from the archives entirely,
+it should not be relied on past its EOL date.
+Therefore ROS 1 must move to Python 3.
+
+Context
+=======
+
+Different versions of Python can't be mixed within a single ROS distribution.
+A package using Python 3 can't import Python modules from a package built using
+Python 2.
+All ROS packages containing Python code need to support Python 3 before they can
+be released into Noetic.
+
+Past proposals for Python 3 support have assumed a ROS distro will need to
+support Python 2 and Python 3 at the same time.
+Since then it was decided the first ROS 1 distribution to support Python 3
+will be Noetic Ninjemys (May 2020) [3]_, and it will only support Python 3.
+This REP only describes the current proposal since the change in requirements
+has made past proposals obsolete.
+
+This proposal does not specify a minimum minor version for Python.
+The minimum version will be defined in REP 3 [4]_ together with all other
+versions.
+
+Proposal
+========
+
+Python 3 support at the time of Noetic release means:
+
+* All ROS tooling supports creating a Python 3 only ROS distro
+* All ROS packages released to Noetic support Python 3
+
+The amount of work to transition to Python 3 is expected to be significant.
+The entire ROS community will need to share the workload.
+This effort should be started as soon as possible to reduce the risk of the
+Noetic release being delayed.
+
+Tooling support for Python 3
+----------------------------
+
+rosdep keys
+'''''''''''
+All existing Python rosdep keys including ``<platform>_python3`` entries will
+stay the same to avoid breaking unknown existing use cases.
+Even if a key refers to a Python 2 package on one platform and Python 3 on
+another, it should stay that way.
+
+Some Python rosdep keys have ``<platform>_python3`` entries.
+This was an experiment at having a rosdep key conditionally evaluate to either
+a Python 2 or Python 3 system dependency.
+No new ``<platform>_python3`` entries should be added, and the entries should
+not be relied on.
+
+For new Python keys, when a platform has separate Python 2 and Python 3 versions
+of a package, there should be a rosdep key for each.
+For example, consider the two keys ``python-catkin-pkg`` and ``python3-catkin-pkg``.
+The ``python-catkin-pkg`` key should resolve to a Python 2 dependency on any
+platform where one exists, while ``python3-catkin-pkg`` should only resolve to a
+Python 3 dependency.
+
+
+ROS_PYTHON_VERSION
+''''''''''''''''''
+
+``ROS_PYTHON_VERSION`` is an environment variable that is set to either ``2`` or
+``3`` to indicate the major version of Python supported by a ROS distro.
+The intent is to allow ``package.xml`` files to have conditional Python
+dependencies.
+
+It is set by the package ``ros_environment`` when sourcing a ROS workspace.
+The version set will be dictated by REP 3 [4]_.
+For example, in Melodic it defaults to ``2``, but in Noetic it must be ``3``.
+
+Users building ROS from source will not have an existing ROS workspace set, so
+the variable won't be set, and conditional dependencies can't be evaluated
+when using ``rosdep`` to install them.
+To fix this, ``rosdep`` should set ``ROS_PYTHON_VERSION`` to the same version of
+Python used to invoke it.
+It will get this information using ``sys.version[0]``.
+In order to use a different Python version, ``ROS_PYTHON_VERSION`` must be set
+explicitly.
+
+rosdep and pip packages
+'''''''''''''''''''''''
+Unlike Debian packages, keys that resolve to packages on PyPI may refer to
+either Python 2 or Python 3 dependencies.
+The version that gets downloaded depends on the version of Python used to
+invoke pip.
+Rosdep currently calls the command ``pip``.
+It might use Python 2 when using the Debian package ``python-pip``, but it could
+use Python 3 inside a venv.
+
+``rosdep`` could default to ``sys.executable -m pip`` to use the same version of
+Python as rosdep, but a user building the ROS 1 bridge for Melodic is likely
+using the Debian package ``python3-rosdep``.
+To support this, rosdep should use ``python$ROS_PYTHON_VERSION -m pip``.
+Since ``rosdep`` sets ``ROS_PYTHON_VERSION`` for itself if unset, this will default
+to the same version of Python rosdep was run with.
+However, it will be overridden automatically when a ROS workspace is sourced.
+
+ROS packages support for Python 3
+---------------------------------
+
+There are many ROS packages using Python that will need to be modified to
+support Python 3.
+Packages using different branches for different ROS distros can drop support
+for Python 2 in their Noetic branch.
+Packages which use the same branch in multiple ROS distros may need to support
+both Python 2 and Python 3 at the same time.
+This section describes what needs to be done in both cases.
+
+Shebangs and reliance on the Python command
+'''''''''''''''''''''''''''''''''''''''''''
+Python scripts on UNIX systems typically have shebang lines written as:
+
+.. code-block: bash
+
+    #!/usr/bin/env python
+
+PEP 394 recommends distributed Python scripts to use either ``python2`` or
+``python3`` [7]_.
+The ``python`` command cannot be trusted to a specific Python version.
+On older ROS distros, scripts can continue to use ``python`` since they're known
+to work on those platforms.
+In preparation for Noetic, these shebang's should be rewritten to the specific
+version of Python supported, ``python3``.
+
+Packages using the same branch will need to conditionally rewrite the shebangs.
+Packages can use the CMake macro ``catkin_install_python()`` to install Python
+scripts with rewritten shebangs.
+
+The same issue appears in scripts that call the ``python`` command directly.
+If they are Python scripts, they should be modified to invoke ``sys.executable``.
+Otherwise, they should be edited to invoke the specific version of Python they
+require, or ``python$ROS_PYTHON_VERSION`` if the script works with both.
+
+Dependencies and package.xml
+''''''''''''''''''''''''''''
+
+On platforms where the target version of Python is 2, the package.xml of a ROS
+package must refer to Python 2 dependencies, and when the target Python
+version is 3 it must refer to Python 3 dependencies.
+Packages which release from different branches for each ROS distro can replace
+rosdep keys that resolve to Python 2 dependencies with ones that resolve to
+Python 3 equivalents.
+Packages using the same code base for multiple ROS distros should instead use
+conditional dependencies as described in REP 149 [5]_.
+
+.. code-block: xml
+
+    <depend condition="$ROS_PYTHON_VERSION == '2'">python-numpy</depend>
+    <depend condition="$ROS_PYTHON_VERSION == '3'">python3-numpy</depend>
+
+If ``ROS_PYTHON_VERSION`` is relied upon at build time, such as when using
+``catkin_install_python()`` to rewrite shebangs, then the package must declare a
+``<buildtool_depend>`` on ``ros_environment``.
+Any ROS package which uses ``ROS_PYTHON_VERSION`` in a script intended to be
+run at runtime should add an ``<exec_depend>`` tag for ``ros_environment``.
+
+Making Python fixes available to downstream packages
+''''''''''''''''''''''''''''''''''''''''''''''''''''
+
+Transitioning to Python 3 is expected to be a significant effort.
+Typically, ROS packages are tested using the ROS build farm; however, that
+will not be available until packages for the targeted Ubuntu distribution
+become available.
+Instead, a placeholder ``Noetic`` ``distribution.yaml`` will be made available in
+advance of the buildfarm availability.
+Maintainers should add ``source`` entries for their Noetic branches to this file
+to enable downstream users to use ``rosinstall_generator`` with the
+``--upstream-development`` flag to get Python 3 fixes.
+Instructions to build from source using Python 3 will be made available to
+the ROS community.
+
+Once the build farm is available, maintainers should release packages with
+Python 3 fixes to Noetic as soon as possible, even if they intend to make
+breaking changes later.
+
+Organizing community effort
+---------------------------
+
+In order to achieve this, prior to the Noetic release community members must
+be able to see:
+
+* which ROS packages already support Python 3
+* which ROS packages need help supporting Python 3
+
+The presence of a ``source`` entry in the Noetic ``distribution.yaml`` should be
+taken to mean a package has started transitioning to Python 3.
+Community members can use the differences between this and the previous ROS
+distro's ``distribution.yaml`` as an indication of which packages would benefit
+the most from their contributions.
+
+There are many ROS package maintainers in the community, and each has the
+responsibility of deciding how the packages they maintain should make the
+transition to Python 3.
+On an individual repository level, community members are encouraged to open
+issues and pull requests with Python 3 fixes.
+
+References
+==========
+
+.. [1] PEP 373 Python 2.7 Release Schedule
+   (https://www.Python.org/dev/peps/pep-0373/)
+.. [2] Python2 to be demoted to universe
+   (https://bugs.launchpad.net/ubuntu/+source/swift/+bug/1817023)
+.. [3] Planning future ROS 1 distributions
+   (https://discourse.ros.org/t/planning-future-ros-1-distribution-s/6538)
+.. [4] REP-0003 Target Platforms
+   (http://ros.org/reps/rep-0003)
+.. [5] REP-0149 Package Manifest Format Three Specification
+   (http://ros.org/reps/rep-0149)
+.. [6] ROS Wiki - Python 2 and 3 compatible code
+   (http://wiki.ros.org/Python_2_and_3_compatible_code)
+.. [7] PEP 394 The "Python" Command on Unix-Like Systems
+   (https://www.Python.org/dev/peps/pep-0394/)
+
+Copyright
+=========
+
+This document has been placed in the public domain.
+
+..
+   Local Variables:
+   mode: indented-text
+   indent-tabs-mode: nil
+   sentence-end-double-space: t
+   fill-column: 70
+   coding: utf-8
+   End: