Difference in binary builds between pip 19.3 and 19.3.1 #7240

iamareebjamal · 2019-10-19T20:23:57Z

Environment

pip version: 19.3.1
Python version: 3.7.4/3.7.5
OS: Linux (alpine)

Description

We use multistage builds to make our docker image compact.

https://github.com/fossasia/open-event-server/blob/b7ba05cfaefdc257e9ae07d264044668dadffce5/Dockerfile

We are using python:3.7-alpine as base image which had pip 19.3 pre-installed.
Installing dependencies with build tools like gcc and others in the initial stage and then copying them to the later stage. This is a common practice and recommended in the docker community.

We installed the build tools and requirements using apt and pip which installed correctly using this command:

pip install --install-option="--prefix=/install" setuptools && \
    LIBRARY_PATH=/lib:/usr/lib pip install --install-option="--prefix=/install" -r /requirements.txt

Then, we copy the installed libraries to the next stage using the command
COPY --from=builder /install /usr/local.

Now, all the libraries installed in the previous stage should be present in the next stage. Which was the case until today.

When we built the release container, it did not have importlib_metadata and threw ModuleError.

After hours of debugging, and jumping through diffs of previously working diff environment, we found that it was not installed at all along with some other libraries. More time consuming debugging later, I found out a difference between environments. The broken environment had Python 3.7.5 and pip 19.3.1 and working one had Python 3.7.4 and pip 19.3.

A quick matrix test on Python 3.7.4 with pip 19.3.1 and Python 3.7.5 with pip 19.3 showed that indeed the problem was with pip and not Python version.

After analyzing the missing dependencies, we found out that the only ones missing were the ones that were built during the previous stage. One of which was the aforementioned importlib_metadata along with several other built libraries.

Expected behavior

pip 19.3.1 which is a patch version should not break backward compatibility and retain the behavior as 19.3.

How to Reproduce

Disable wheel packages like in alpine and build the libraries
Copy from one install to another
Find the library to be missing

OR

Use the above-linked dockerfile.
Build it using this command - docker build . -t eventyay/open-event-server
Run the shell using this command - docker run -it --entrypoint /bin/sh eventyay/open-event-server -s
Run pip freeze | grep import
Change the version from python:3.7-alpine to python:3.7.4-alpine or add pip install pip==19.3 in the first RUN step to lock pip, and repeat steps 2-4 and you'll find the library

Output

import importlib_metadata
ModuleNotFoundError: No module named 'importlib_metadata'

The text was updated successfully, but these errors were encountered:

pypa/pip#7240

xavfernandez · 2019-10-19T21:12:40Z

Hello @iamareebjamal,

If I understand correctly your Dockerfile basically installs https://github.com/fossasia/open-event-server/blob/b7ba05cfaefdc257e9ae07d264044668dadffce5/requirements/common.txt and nothing in there depends on importlib_metadata so having it installed was a bug that was fixed in the patch version ?

iamareebjamal · 2019-10-19T21:14:27Z

No, the problem is not related to importlib_metadata. You change importlib_metadata to any library which is built during install

Also, importlib_metadata is a transitive dependency of several of our dependencies.

I can provide a pipdeptree if it helps. Many direct dependencies weren't installed as well

xavfernandez · 2019-10-19T21:16:58Z

I can provide a pipdeptree if it helps.

It would 👍

iamareebjamal · 2019-10-19T21:26:41Z

https://www.diffchecker.com/ryCo05HG

19.3

info

APScheduler==3.6.1
  - pytz [required: Any, installed: 2019.3]
  - setuptools [required: >=0.7, installed: 41.4.0]
  - six [required: >=1.4.0, installed: 1.12.0]
  - tzlocal [required: >=1.2, installed: 2.0.0]
    - pytz [required: Any, installed: 2019.3]
arrow==0.15.2
  - python-dateutil [required: Any, installed: 2.8.0]
    - six [required: >=1.5, installed: 1.12.0]
bleach==3.1.0
  - six [required: >=1.9.0, installed: 1.12.0]
  - webencodings [required: Any, installed: 0.5.1]
blinker==1.4
diff-match-patch==20181111
elasticsearch-dsl==7.0.0
  - elasticsearch [required: >=7.0.0,<8.0.0, installed: 7.0.5]
    - urllib3 [required: >=1.21.1, installed: 1.25.6]
  - python-dateutil [required: Any, installed: 2.8.0]
    - six [required: >=1.5, installed: 1.12.0]
  - six [required: Any, installed: 1.12.0]
envparse==0.2.0
eventlet==0.25.1
  - dnspython [required: >=1.15.0, installed: 1.16.0]
  - greenlet [required: >=0.3, installed: 0.4.15]
  - monotonic [required: >=1.4, installed: 1.5]
  - six [required: >=1.10.0, installed: 1.12.0]
factory-boy==2.12.0
  - Faker [required: >=0.7.0, installed: 2.0.3]
    - python-dateutil [required: >=2.4, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.12.0]
    - six [required: >=1.10, installed: 1.12.0]
    - text-unidecode [required: ==1.3, installed: 1.3]
Flask-Admin==1.5.4
  - Flask [required: >=0.7, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - wtforms [required: Any, installed: 2.2.1]
Flask-Caching==1.7.2
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-CeleryExt==0.3.2
  - celery [required: >=3.1, installed: 4.3.0]
    - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0]
    - kombu [required: >=4.4.0,<5.0, installed: 4.6.5]
      - amqp [required: ==2.5.1, installed: 2.5.1]
        - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0]
      - importlib-metadata [required: >=0.18, installed: 0.23]
        - zipp [required: >=0.5, installed: 0.6.0]
          - more-itertools [required: Any, installed: 7.2.0]
    - pytz [required: >dev, installed: 2019.3]
    - vine [required: >=1.3.0, installed: 1.3.0]
  - celery [required: >=4.3, installed: 4.3.0]
    - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0]
    - kombu [required: >=4.4.0,<5.0, installed: 4.6.5]
      - amqp [required: ==2.5.1, installed: 2.5.1]
        - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0]
      - importlib-metadata [required: >=0.18, installed: 0.23]
        - zipp [required: >=0.5, installed: 0.6.0]
          - more-itertools [required: Any, installed: 7.2.0]
    - pytz [required: >dev, installed: 2019.3]
    - vine [required: >=1.3.0, installed: 1.3.0]
  - Flask [required: >=0.10, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Cors==3.0.8
  - Flask [required: >=0.9, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - Six [required: Any, installed: 1.12.0]
Flask-Elasticsearch==0.2.5
  - Elasticsearch [required: Any, installed: 7.0.5]
    - urllib3 [required: >=1.21.1, installed: 1.25.6]
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-JWT-Extended==3.23.0
  - Flask [required: >=1.0, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - PyJWT [required: >=1.6.4, installed: 1.7.1]
  - six [required: Any, installed: 1.12.0]
  - Werkzeug [required: >=0.14, installed: 0.16.0]
Flask-Limiter==1.1.0
  - Flask [required: >=0.8, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - limits [required: Any, installed: 1.3]
    - six [required: >=1.4.1, installed: 1.12.0]
  - six [required: >=1.4.1, installed: 1.12.0]
Flask-Login==0.4.1
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Migrate==2.5.2
  - alembic [required: >=0.7, installed: 1.2.1]
    - Mako [required: Any, installed: 1.1.0]
      - MarkupSafe [required: >=0.9.2, installed: 1.1.1]
    - python-dateutil [required: Any, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.12.0]
    - python-editor [required: >=0.3, installed: 1.0.4]
    - SQLAlchemy [required: >=1.1.0, installed: 1.3.10]
  - Flask [required: >=0.9, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - Flask-SQLAlchemy [required: >=1.0, installed: 2.4.1]
    - Flask [required: >=0.10, installed: 1.1.1]
      - click [required: >=5.1, installed: 7.0]
      - itsdangerous [required: >=0.24, installed: 1.1.0]
      - Jinja2 [required: >=2.10.1, installed: 2.10.3]
        - MarkupSafe [required: >=0.23, installed: 1.1.1]
      - Werkzeug [required: >=0.15, installed: 0.16.0]
    - SQLAlchemy [required: >=0.8.0, installed: 1.3.10]
flask-redis==0.4.0
  - Flask [required: >=0.8, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - redis [required: >=2.7.6, installed: 3.3.11]
Flask-REST-JSONAPI==0.12.6
  - Flask [required: >=0.11, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - marshmallow [required: >=2.15.1,<3, installed: 2.15.6]
  - marshmallow-jsonapi [required: Any, installed: 0.22.0]
    - marshmallow [required: >=2.15.2, installed: 2.15.6]
  - six [required: Any, installed: 1.12.0]
  - sqlalchemy [required: Any, installed: 1.3.10]
Flask-Script==2.0.6
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Scrypt==0.1.3.6
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - scrypt [required: Any, installed: 0.8.13]
forex-python==1.5
  - requests [required: Any, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
  - simplejson [required: Any, installed: 3.16.0]
geoip2==2.9.0
  - maxminddb [required: >=1.4.0, installed: 1.5.1]
  - requests [required: >=2.9, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
google-compute-engine==2.8.13
  - boto [required: Any, installed: 2.49.0]
  - distro [required: Any, installed: 1.4.0]
  - setuptools [required: Any, installed: 41.4.0]
gunicorn==19.9.0
healthcheck==1.3.3
humanize==0.5.1
icalendar==4.0.3
  - python-dateutil [required: Any, installed: 2.8.0]
    - six [required: >=1.5, installed: 1.12.0]
  - pytz [required: Any, installed: 2019.3]
marrow.mailer==4.0.3
  - marrow.util [required: <2.0, installed: 1.2.3]
oauth2==1.9.0.post1
  - httplib2 [required: Any, installed: 0.14.0]
omise==0.8.1
  - requests [required: >=2.12.1, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
paypalrestsdk==1.13.1
  - pyopenssl [required: >=0.15, installed: 19.0.0]
    - cryptography [required: >=2.3, installed: 2.8]
      - cffi [required: >=1.8,!=1.11.3, installed: 1.13.0]
        - pycparser [required: Any, installed: 2.19]
      - six [required: >=1.4.1, installed: 1.12.0]
    - six [required: >=1.5.2, installed: 1.12.0]
  - requests [required: >=1.0.0, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
  - six [required: >=1.0.0, installed: 1.12.0]
pipdeptree==0.13.2
  - pip [required: >=6.0.0, installed: 19.3]
psycopg2-binary==2.8.3
pycountry==19.8.18
pycryptodome==3.9.0
python-dotenv==0.10.3
python-geoip-geolite2==2015.303
  - python-geoip [required: Any, installed: 1.2]
python-magic==0.4.15
python-pentabarf-xml==0.19
PyYAML==5.1.2
qrcode==6.1
  - six [required: Any, installed: 1.12.0]
requests-oauthlib==1.2.0
  - oauthlib [required: >=3.0.0, installed: 3.1.0]
  - requests [required: >=2.0.0, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
sendgrid==6.1.0
  - python-http-client [required: >=3.2.1, installed: 3.2.1]
sentry-sdk==0.13.0
  - certifi [required: Any, installed: 2019.9.11]
  - urllib3 [required: >=1.9, installed: 1.25.6]
SQLAlchemy-Continuum==1.3.9
  - SQLAlchemy [required: >=1.0.8, installed: 1.3.10]
  - SQLAlchemy-Utils [required: >=0.30.12, installed: 0.34.2]
    - six [required: Any, installed: 1.12.0]
    - SQLAlchemy [required: >=1.0, installed: 1.3.10]
stripe==2.37.2
  - requests [required: >=2.20, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
unicode-slugify==0.1.3
  - six [required: Any, installed: 1.12.0]
  - unidecode [required: Any, installed: 1.1.1]
wheel==0.33.6
xhtml2pdf==0.2.3
  - html5lib [required: >=1.0, installed: 1.0.1]
    - six [required: >=1.9, installed: 1.12.0]
    - webencodings [required: Any, installed: 0.5.1]
  - Pillow [required: Any, installed: 6.2.0]
  - pyPdf2 [required: Any, installed: 1.26.0]
  - reportlab [required: >=3.0, installed: 3.5.31]
    - pillow [required: >=4.0.0, installed: 6.2.0]
  - six [required: Any, installed: 1.12.0]

19.3.1

Warning!!! Possibly conflicting dependencies found:
* pyOpenSSL==19.0.0
 - cryptography [required: >=2.3, installed: ?]
* kombu==4.6.5
 - importlib-metadata [required: >=0.18, installed: ?]
* icalendar==4.0.3
 - python-dateutil [required: Any, installed: ?]
* Faker==2.0.3
 - python-dateutil [required: >=2.4, installed: ?]
* elasticsearch-dsl==7.0.0
 - python-dateutil [required: Any, installed: ?]
* arrow==0.15.2
 - python-dateutil [required: Any, installed: ?]
* alembic==1.2.1
 - python-dateutil [required: Any, installed: ?]
------------------------------------------------------------------------
APScheduler==3.6.1
  - pytz [required: Any, installed: 2019.3]
  - setuptools [required: >=0.7, installed: 41.4.0]
  - six [required: >=1.4.0, installed: 1.12.0]
  - tzlocal [required: >=1.2, installed: 2.0.0]
    - pytz [required: Any, installed: 2019.3]
arrow==0.15.2
  - python-dateutil [required: Any, installed: ?]
bleach==3.1.0
  - six [required: >=1.9.0, installed: 1.12.0]
  - webencodings [required: Any, installed: 0.5.1]
blinker==1.4
cffi==1.13.0
  - pycparser [required: Any, installed: 2.19]
diff-match-patch==20181111
elasticsearch-dsl==7.0.0
  - elasticsearch [required: >=7.0.0,<8.0.0, installed: 7.0.5]
    - urllib3 [required: >=1.21.1, installed: 1.25.6]
  - python-dateutil [required: Any, installed: ?]
  - six [required: Any, installed: 1.12.0]
envparse==0.2.0
eventlet==0.25.1
  - dnspython [required: >=1.15.0, installed: 1.16.0]
  - greenlet [required: >=0.3, installed: 0.4.15]
  - monotonic [required: >=1.4, installed: 1.5]
  - six [required: >=1.10.0, installed: 1.12.0]
factory-boy==2.12.0
  - Faker [required: >=0.7.0, installed: 2.0.3]
    - python-dateutil [required: >=2.4, installed: ?]
    - six [required: >=1.10, installed: 1.12.0]
    - text-unidecode [required: ==1.3, installed: 1.3]
Flask-Admin==1.5.4
  - Flask [required: >=0.7, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - wtforms [required: Any, installed: 2.2.1]
Flask-Caching==1.7.2
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-CeleryExt==0.3.2
  - celery [required: >=3.1, installed: 4.3.0]
    - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0]
    - kombu [required: >=4.4.0,<5.0, installed: 4.6.5]
      - amqp [required: ==2.5.1, installed: 2.5.1]
        - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0]
      - importlib-metadata [required: >=0.18, installed: ?]
    - pytz [required: >dev, installed: 2019.3]
    - vine [required: >=1.3.0, installed: 1.3.0]
  - celery [required: >=4.3, installed: 4.3.0]
    - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0]
    - kombu [required: >=4.4.0,<5.0, installed: 4.6.5]
      - amqp [required: ==2.5.1, installed: 2.5.1]
        - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0]
      - importlib-metadata [required: >=0.18, installed: ?]
    - pytz [required: >dev, installed: 2019.3]
    - vine [required: >=1.3.0, installed: 1.3.0]
  - Flask [required: >=0.10, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Cors==3.0.8
  - Flask [required: >=0.9, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - Six [required: Any, installed: 1.12.0]
Flask-Elasticsearch==0.2.5
  - Elasticsearch [required: Any, installed: 7.0.5]
    - urllib3 [required: >=1.21.1, installed: 1.25.6]
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-JWT-Extended==3.23.0
  - Flask [required: >=1.0, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - PyJWT [required: >=1.6.4, installed: 1.7.1]
  - six [required: Any, installed: 1.12.0]
  - Werkzeug [required: >=0.14, installed: 0.16.0]
Flask-Limiter==1.1.0
  - Flask [required: >=0.8, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - limits [required: Any, installed: 1.3]
    - six [required: >=1.4.1, installed: 1.12.0]
  - six [required: >=1.4.1, installed: 1.12.0]
Flask-Login==0.4.1
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Migrate==2.5.2
  - alembic [required: >=0.7, installed: 1.2.1]
    - Mako [required: Any, installed: 1.1.0]
      - MarkupSafe [required: >=0.9.2, installed: 1.1.1]
    - python-dateutil [required: Any, installed: ?]
    - python-editor [required: >=0.3, installed: 1.0.4]
    - SQLAlchemy [required: >=1.1.0, installed: 1.3.10]
  - Flask [required: >=0.9, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - Flask-SQLAlchemy [required: >=1.0, installed: 2.4.1]
    - Flask [required: >=0.10, installed: 1.1.1]
      - click [required: >=5.1, installed: 7.0]
      - itsdangerous [required: >=0.24, installed: 1.1.0]
      - Jinja2 [required: >=2.10.1, installed: 2.10.3]
        - MarkupSafe [required: >=0.23, installed: 1.1.1]
      - Werkzeug [required: >=0.15, installed: 0.16.0]
    - SQLAlchemy [required: >=0.8.0, installed: 1.3.10]
Flask-REST-JSONAPI==0.12.6
  - Flask [required: >=0.11, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - marshmallow [required: >=2.15.1,<3, installed: 2.15.6]
  - marshmallow-jsonapi [required: Any, installed: 0.22.0]
    - marshmallow [required: >=2.15.2, installed: 2.15.6]
  - six [required: Any, installed: 1.12.0]
  - sqlalchemy [required: Any, installed: 1.3.10]
Flask-Script==2.0.6
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Scrypt==0.1.3.6
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - scrypt [required: Any, installed: 0.8.13]
forex-python==1.5
  - requests [required: Any, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
  - simplejson [required: Any, installed: 3.16.0]
geoip2==2.9.0
  - maxminddb [required: >=1.4.0, installed: 1.5.1]
  - requests [required: >=2.9, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
google-compute-engine==2.8.13
  - boto [required: Any, installed: 2.49.0]
  - distro [required: Any, installed: 1.4.0]
  - setuptools [required: Any, installed: 41.4.0]
gunicorn==19.9.0
healthcheck==1.3.3
humanize==0.5.1
icalendar==4.0.3
  - python-dateutil [required: Any, installed: ?]
  - pytz [required: Any, installed: 2019.3]
marrow.mailer==4.0.3
  - marrow.util [required: <2.0, installed: 1.2.3]
more-itertools==7.2.0
oauth2==1.9.0.post1
  - httplib2 [required: Any, installed: 0.14.0]
omise==0.8.1
  - requests [required: >=2.12.1, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
paypalrestsdk==1.13.1
  - pyopenssl [required: >=0.15, installed: 19.0.0]
    - cryptography [required: >=2.3, installed: ?]
    - six [required: >=1.5.2, installed: 1.12.0]
  - requests [required: >=1.0.0, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
  - six [required: >=1.0.0, installed: 1.12.0]
pipdeptree==0.13.2
  - pip [required: >=6.0.0, installed: 19.3.1]
psycopg2-binary==2.8.3
pycountry==19.8.18
pycryptodome==3.9.0
python-dotenv==0.10.3
python-geoip-geolite2==2015.303
  - python-geoip [required: Any, installed: 1.2]
python-magic==0.4.15
python-pentabarf-xml==0.19
PyYAML==5.1.2
qrcode==6.1
  - six [required: Any, installed: 1.12.0]
redis==3.3.11
requests-oauthlib==1.2.0
  - oauthlib [required: >=3.0.0, installed: 3.1.0]
  - requests [required: >=2.0.0, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
sendgrid==6.1.0
  - python-http-client [required: >=3.2.1, installed: 3.2.1]
sentry-sdk==0.13.0
  - certifi [required: Any, installed: 2019.9.11]
  - urllib3 [required: >=1.9, installed: 1.25.6]
SQLAlchemy-Continuum==1.3.9
  - SQLAlchemy [required: >=1.0.8, installed: 1.3.10]
  - SQLAlchemy-Utils [required: >=0.30.12, installed: 0.34.2]
    - six [required: Any, installed: 1.12.0]
    - SQLAlchemy [required: >=1.0, installed: 1.3.10]
stripe==2.37.2
  - requests [required: >=2.20, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
unicode-slugify==0.1.3
  - six [required: Any, installed: 1.12.0]
  - unidecode [required: Any, installed: 1.1.1]
wheel==0.33.6
xhtml2pdf==0.2.3
  - html5lib [required: >=1.0, installed: 1.0.1]
    - six [required: >=1.9, installed: 1.12.0]
    - webencodings [required: Any, installed: 0.5.1]
  - Pillow [required: Any, installed: 6.2.0]
  - pyPdf2 [required: Any, installed: 1.26.0]
  - reportlab [required: >=3.0, installed: 3.5.31]
    - pillow [required: >=4.0.0, installed: 6.2.0]
  - six [required: Any, installed: 1.12.0]

pradyunsg · 2019-10-20T04:02:02Z

@iamareebjamal Hi! I've edited your bug report to read more easily (shorter sentences, imperative tone) and to tone down some of the wording used.

Using all caps (equivalent to shouting via text) isn't a good move when you're asking for help from volunteers who do this stuff because they find it fun. :)

If you have any concerns with that, please file a new issue on pip or drop me an email on {username}@gmail.com.

pradyunsg · 2019-10-20T04:14:02Z

Looking at the diff between the release tags: 19.3...19.3.1

I think what broke your usecase was the fact that we fixed a major bug in PEP 517 based builds. I'd have liked to have made that fix in 19.3 but I didn't have the bandwidth to and AFAICT, that fix was fairly innocuous. I'm not sure there's anything actionable now for pip, other than acknowledging that cached docker containers that were built earlier, should be invalidated since a bug in pip's build process was fixed which affects how we can use the cache.

Sorry if that sounds a little bleak but well, pip's used in so many ways that at this point, every non-refactoring patch breaks something, usually in ways that no one thought of when working on this stuff. 🤷🏻‍♂️

chrahunt · 2019-10-20T04:20:04Z

@iamareebjamal, in your Dockerfile please pass an explicit --prefix=/install argument to pip itself instead of --install-options='--prefix=/install'. Pip will internally update --install-options as well as take other necessary measures to ensure installed packages respect the --prefix argument. I tested this locally and the resulting image (containing dependencies installed with pip 19.3.1) had the same dependencies as the image built with dependencies installed with 19.3. You'll probably also want to pass --no-warn-script-location to reduce some of the noise which is irrelevant to your use case.

There was a bug fixed in #6606 (released in 19.3.1) as mentioned by @pradyunsg that now correctly builds projects that use PEP 517 without implicitly going through setuptools. This fix was causing an issue in your case because the packages that were now using PEP 517 were not picking up the prefix directory passed via --install-options.

iamareebjamal · 2019-10-20T06:30:34Z

Sorry if it felt like I was using caps to emphasize or shout. I was simply referring to MAJOR.MINOR.PATCH in semver context, not for shouting

iamareebjamal · 2019-10-20T06:34:23Z

@chrahunt If pip indeed follows semantic versioning, then its behaviour should not change between patch updates. I can add the fix from my end for sure, but this is considered a breaking change, and hence should be communicated and moved to a minor release, rather than a patch one. I understand that a major release will not be practical, but even minor releases warrant backward compatibility with feature additions. And patch updates should only fix bugs without compromising any previous behaviour

About action that pip can take, I don't know if this would be considered rude or impossible as well, but most other tool authors create a new patch version with behaviour restored to surprise the least amount of users, and then introduce the fix with a minor update

Reference: stripe/stripe-python#582

pfmoore · 2019-10-20T07:02:00Z

@iamareebjamal pip does not follow semantic versioning, we use calendar based versioning. See the documentation for details. Specifically, version 19.3.1 is the first bugfix release of the 3rd release in 2019.

iamareebjamal · 2019-10-20T07:11:23Z

Well, that's sad :( Documentation update of 19.3.1 won't notify people of the issue. And they may spend days debugging the issue and may not even be able to source it to pip. This will cause a waste of time and stress. I know something might break in any patch version of pip because of its vast use case surface area, but the next section in the documentation you linked talks about Deprecation Policy

It says that pip will start issuing warnings for 6 months before changing public visible behaviour. https://pip.pypa.io/en/latest/development/release-process/#deprecation-policy

That is awesome! Because it is stricter than semantic versioning and would align with pip's own guidelines as well. And also follows the principle of least surprises.
Now, it may happen that this behaviour was not documented, but it is widely used. If the policy is followed and respected, it's great for both pip and its users.
If not, then it is a dependency hell :( https://semver.org/#introduction

chrahunt · 2019-10-21T21:36:00Z

Hi @iamareebjamal. One thing I think would be completely reasonable (and that we've done in the past) is to make (and pin, if common enough) a canonical issue to communicate the best information we have on how to identify and remedy the problem. This should get picked up by search engines too, reducing any burden.

To help in creating that, can you provide information on the origin of --install-option in your Dockerfile? Another issue #7253 seems similar enough that maybe they are from the same place (blog post, SO post, etc) and maybe we could take some wording from that or change it at the source.

iamareebjamal · 2019-10-21T22:12:21Z

Sounds good enough. The maintainer who added the --install-option flag is no longer available but I'll try to contact him and search on my own as well for articles mentioning this behavior.

Thanks for the help

Also, change to new syntax of pip for install prefix Ref: #6543 pypa/pip#7240 (comment)

Also, change to new syntax of pip for install prefix Ref: fossasia#6543 pypa/pip#7240 (comment)

hugovk · 2022-05-05T15:57:44Z

Is this issue still valid or can it be closed?

pradyunsg · 2022-05-05T18:15:23Z

This can be closed!

triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Oct 19, 2019

iamareebjamal added a commit to fossasia/open-event-server that referenced this issue Oct 19, 2019

chore: Lock python version in docker to workaround pip

298ce5e

pypa/pip#7240

pradyunsg changed the title ~~PATCH upgrade of 19.3.1 breaks binary builds~~ Difference in binary builds between pip 19.3 and 19.3.1 Oct 20, 2019

chrahunt added S: awaiting response Waiting for a response/more information type: support User Support labels Oct 20, 2019

triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label Oct 20, 2019

no-response bot removed the S: awaiting response Waiting for a response/more information label Oct 20, 2019

chrahunt mentioned this issue Oct 21, 2019

pip 19.3.1 isn't respecting --install-option="--prefix=/install" for some packages #7253

Closed

chrahunt added the S: awaiting response Waiting for a response/more information label Oct 21, 2019

no-response bot removed the S: awaiting response Waiting for a response/more information label Oct 21, 2019

chrahunt mentioned this issue Nov 7, 2019

Disallow explicitly passing install-location-related arguments in --install-options #7309

Closed

iamareebjamal mentioned this issue Nov 28, 2019

chore: Remove cruft and simplify Dockerfile fossasia/open-event-server#6633

Merged

iamareebjamal added a commit to fossasia/open-event-server that referenced this issue Nov 28, 2019

chore: Remove cruft and simplify Dockerfile (#6633)

d8a71db

Also, change to new syntax of pip for install prefix Ref: #6543 pypa/pip#7240 (comment)

codedsun pushed a commit to codedsun/open-event-server that referenced this issue Dec 22, 2019

chore: Remove cruft and simplify Dockerfile (fossasia#6633)

1784182

Also, change to new syntax of pip for install prefix Ref: fossasia#6543 pypa/pip#7240 (comment)

iamareebjamal mentioned this issue Oct 13, 2020

chore(docs): Improve local installation guide fossasia/open-event-server#7320

Merged

6 tasks

pradyunsg closed this as completed May 5, 2022

github-actions bot locked as resolved and limited conversation to collaborators Jun 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Difference in binary builds between pip 19.3 and 19.3.1 #7240

Difference in binary builds between pip 19.3 and 19.3.1 #7240

iamareebjamal commented Oct 19, 2019 •

edited by pradyunsg

Loading

xavfernandez commented Oct 19, 2019 •

edited by pradyunsg

Loading

iamareebjamal commented Oct 19, 2019 •

edited

Loading

xavfernandez commented Oct 19, 2019

iamareebjamal commented Oct 19, 2019 •

edited by chrahunt

Loading

pradyunsg commented Oct 20, 2019

pradyunsg commented Oct 20, 2019 •

edited

Loading

chrahunt commented Oct 20, 2019 •

edited

Loading

iamareebjamal commented Oct 20, 2019

iamareebjamal commented Oct 20, 2019 •

edited

Loading

pfmoore commented Oct 20, 2019

iamareebjamal commented Oct 20, 2019 •

edited

Loading

chrahunt commented Oct 21, 2019

iamareebjamal commented Oct 21, 2019

hugovk commented May 5, 2022

pradyunsg commented May 5, 2022

Difference in binary builds between pip 19.3 and 19.3.1 #7240

Difference in binary builds between pip 19.3 and 19.3.1 #7240

Comments

iamareebjamal commented Oct 19, 2019 • edited by pradyunsg Loading

xavfernandez commented Oct 19, 2019 • edited by pradyunsg Loading

iamareebjamal commented Oct 19, 2019 • edited Loading

xavfernandez commented Oct 19, 2019

iamareebjamal commented Oct 19, 2019 • edited by chrahunt Loading

pradyunsg commented Oct 20, 2019

pradyunsg commented Oct 20, 2019 • edited Loading

chrahunt commented Oct 20, 2019 • edited Loading

iamareebjamal commented Oct 20, 2019

iamareebjamal commented Oct 20, 2019 • edited Loading

pfmoore commented Oct 20, 2019

iamareebjamal commented Oct 20, 2019 • edited Loading

chrahunt commented Oct 21, 2019

iamareebjamal commented Oct 21, 2019

hugovk commented May 5, 2022

pradyunsg commented May 5, 2022

iamareebjamal commented Oct 19, 2019 •

edited by pradyunsg

Loading

xavfernandez commented Oct 19, 2019 •

edited by pradyunsg

Loading

iamareebjamal commented Oct 19, 2019 •

edited

Loading

iamareebjamal commented Oct 19, 2019 •

edited by chrahunt

Loading

pradyunsg commented Oct 20, 2019 •

edited

Loading

chrahunt commented Oct 20, 2019 •

edited

Loading

iamareebjamal commented Oct 20, 2019 •

edited

Loading

iamareebjamal commented Oct 20, 2019 •

edited

Loading