Compiled source files are not identical across installations even when SOURCE_DATE_EPOCH is set to the same value

[dshcherb]

Environment

• pip version: 20.0.2
• Python version: 3.8.2
• OS: Ubuntu 20.04

Description

.pyc files generated during wheel installation include absolute paths of files unpacked to temporary directories such as '/tmp/pip-unpacked-wheel-79blnlyq/setuptools/command/bdist_egg.py.

Thus, even when SOURCE_DATE_EPOCH is set to the same value across different installations of the same package, .pyc files are different.

A distro bug with more details: https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1882535 starting with comment 2.

This happens because pip passes absolute paths for source files to py_compile which then uses source_to_code to get a code object that includes this absolute file path. py_compile then uses marshal.dumps on that code object to generate the final .pyc file - as a result, the .pyc file content always includes bytes from the temporary path generated by pip => builds aren't reproducible even when the PycInvalidationMode is forced to CHECKED_HASH via SOURCE_DATE_EPOCH.

https://github.com/python/cpython/blob/0f5a28f834bdac2da8a04597dc0fc5b71e50da9d/Lib/py_compile.py#L144-L145
https://docs.python.org/3/library/importlib.html#importlib.abc.InspectLoader.source_to_code

Expected behavior

.pyc files have the same hash across installations if SOURCE_DATE_EPOCH is the same.

How to Reproduce

export SOURCE_DATE_EPOCH=1591640328
pip3 install -t ./ setuptools

sha256sum setuptools/command/__pycache__/bdist_egg.cpython-38.pyc
e52f3e2bb06005c3576fcc7402f18e5757cc2c53fc39b5f366c8ea159191c136  setuptools/command/__pycache__/bdist_egg.cpython-38.pyc

grep -a /tmp/pip-unpacked-wheel setuptools/command/__pycache__/bdist_egg.cpython-38.pyc
EntryPoint)�Library)�Command�SetuptoolsDeprecationWarning)get_path�get_python_versioncCtd�S)N�purelib)r�rr�@/tmp/pip-unpacked-wheel-lbfyixdd/setuptools/command/bdist_egg.py�
       _get_purelibr)�get_python_librcCtd�S)NF)rrrrrr scCs2d|krtj�|�d}|�d�r.|dd�}|S)N�.r�modulei����)�os�pathsplitextendswith)filenamerrr�
                                                                                                                                          strip_module$s


rm -r ./*

sha256sum setuptools/command/__pycache__/bdist_egg.cpython-38.pyc
54f4a618de763a70b6d055de478735e3e7f6f202ecf5706f5b44bfbddf8c1fd5  setuptools/command/__pycache__/bdist_egg.cpython-38.pyc

grep -a /tmp/pip-unpacked-wheel setuptools/command/__pycache__/bdist_egg.cpython-38.pyc
EntryPoint)�Library)�Command�SetuptoolsDeprecationWarning)get_path�get_python_versioncCtd�S)N�purelib)r�rr�@/tmp/pip-unpacked-wheel-3x647bdf/setuptools/command/bdist_egg.py�
       _get_purelibr)�get_python_librcCtd�S)NF)rrrrrr scCs2d|krtj�|�d}|�d�r.|dd�}|S)N�.r�modulei����)�os�pathsplitextendswith)filenamerrr�
                                                                                                                                          strip_module$s

[FRidh]

This is a duplicate of #7808.

[pradyunsg]

Thanks @FRidh! Closing as a duplicate of #7808.