Airflow installation fails when with latest `setuptools` #18075

edgarrmondragon · 2021-09-08T01:29:18Z

Apache Airflow version

2.1.3

Operating System

Ubuntu 20.04

Versions of Apache Airflow Providers

NA

Deployment

Other Docker-based deployment

Deployment details

Ubuntu Docker image, Airflow installed inside a virtualenv in the image.

What happened

Trying to pip-install Airflow in a virtualenv where setuptools is >=58.0.2 fails:

Orchestrator 'airflow' could not be installed: failed to install plugin 'airflow'.
    ERROR: Command errored out with exit status 1:
     command: /project/.meltano/orchestrators/airflow/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-g8rc5i_t/flask-openid_b02636d256a1457daa8e5d35116652ca/setup.py'"'"'; __file__='"'"'/tmp/pip-install-g8rc5i_t/flask-openid_b02636d256a1457daa8e5d35116652ca/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-r8lhwbha
         cwd: /tmp/pip-install-g8rc5i_t/flask-openid_b02636d256a1457daa8e5d35116652ca/
    Complete output (1 lines):
    error in Flask-OpenID setup command: use_2to3 is invalid.

The setuptools team deprecated use_2to3 (see pypa/setuptools#2086) recently and this is breaking installation of packages which rely on that legacy key. In the case of Airflow, Flask-OpenID, required by Flask-AppBuilder, is the offending dependency. See https://github.com/mitsuhiko/flask-openid/blob/v1.2.5/setup.py#L32.

This should probably be fixed by that project's maintainers, but we (Meltano) thought it was a good idea to file this issue here in case Airflow users run into the problem. Just removing use_2to3 seems to work 😃.

What you expected to happen

Airflow's dependencies should be prepared for new versions of setuptools and Python (3.10).

How to reproduce

$ python -m venv venv
$ source venv/bin/activate
(venv) $ pip install "setuptools==58.0.2"
(venv) $ pip install apache-airflow
...
Collecting Flask-OpenID<2,>=1.2.5
  Using cached Flask-OpenID-1.2.5.tar.gz (43 kB)
    ERROR: Command errored out with exit status 1:
     command: python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/8b/z9sdckf92_g3hgnh6nv0yhbh0000gn/T/pip-install-wazcqi5t/flask-openid_d71a670325d74cf6b46f7347ab3e06f4/setup.py'"
'"'; __file__='"'"'/private/var/folders/8b/z9sdckf92_g3hgnh6nv0yhbh0000gn/T/pip-install-wazcqi5t/flask-openid_d71a670325d74cf6b46f7347ab3e06f4/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringI
O('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/8b/z9sdckf92_g3hgnh6nv0yhbh0000gn/T/pip-pip-egg-inf
o-on8iyf31
         cwd: /private/var/folders/8b/z9sdckf92_g3hgnh6nv0yhbh0000gn/T/pip-install-wazcqi5t/flask-openid_d71a670325d74cf6b46f7347ab3e06f4/
    Complete output (1 lines):
    error in Flask-OpenID setup command: use_2to3 is invalid

Anything else

The current workaround it to make sure a version of setuptools prior to 58.0.0 is installed in the active Python environment.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

boring-cyborg · 2021-09-08T01:29:20Z

Thanks for opening your first issue here! Be sure to follow the issue template!

uranusjr · 2021-09-08T01:37:29Z

Looks like this should be submitted to pallets-eco/flask-openid#59 instead.

potiuk · 2021-09-08T09:28:44Z

Unfortunately this is one of the things that is not possible to handle in Airflow in already released versions, as the right version of setuptools need to be installed before airflow installation and related packages are attempted to be installed.

However if @mitsuhiko fixes it and release flask-openid without 2_to_3, we will be able to re-pin our constraints to use that new version of flask-openid in our constraints for all released versions of Airflow.

So we really depend on this.

potiuk · 2021-09-08T09:29:14Z

For now - workaround is to use setuptools < 58.0.2

potiuk · 2021-09-08T09:40:12Z

We've figured a possible automated workaround with @uranusjr . Stay tuned.

RafayAK · 2021-09-08T10:12:59Z

@potiuk looking forward to this

RafayAK · 2021-09-08T10:52:45Z

My current workaround in my dockerfile:

# install Airflow with constraints file for compatibility
ARG AIRFLOW_VERSION=2.0.1
ARG PYTHON_VER=3.7
ARG CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VER}.txt"
RUN pip3 install --upgrade pip && \
    pip install -U pip setuptools==58.0.1 wheel \
    && pip install "apache-airflow[statsd]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}" \
    && pip install pyOpenSSL \
    && pip install ffmpeg-python

potiuk · 2021-09-08T10:58:14Z

Yep. For Dockerfile it's easy. But we have a better workaround. Just tested and it works. the problem is that it also broke all Airflow 2 installations which people already use and any attempt to install those, will fail. We have a fix that will make the "official installation method" works for all Airflow 2 releases.

potiuk · 2021-09-08T11:12:15Z

All right - 2.1.3 should now work when you install it with constraints with latest setuptools:

pip install "apache-airflow==2.1.3" --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.1.3/constraints-3.6.txt

Can someone confirm it ? then I will apply the same fix to other released versions and carry on with fixing it for main as well

RafayAK · 2021-09-08T11:13:58Z

All right - 2.1.3 should now work when you install it with constraints with latest setuptools:

pip install "apache-airflow==2.1.3" --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.1.3/constraints-3.6.txt

Can someone confirm it ? then I will apply the same fix to other released versions and carry on with fixing it for main as well

let me try this

potiuk · 2021-09-08T11:14:47Z

Fr those curious ones - I'we built Flask-OpenId 1.2.6 with removed 2_to_3, we host it here: https://github.com/apache/airflow-flask-openid-fork/releases/tag/v1.2.6 and our constraints have been updated to install it from there:

64508c7

RafayAK · 2021-09-08T11:25:18Z

@potiuk just tried building my dockerfile again with the following:

# install Airflow with constraints file for compatibility
ARG AIRFLOW_VERSION=2.0.1
ARG PYTHON_VER=3.7
ARG CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VER}.txt"
RUN pip3 install --upgrade pip && \
    pip install -U pip setuptools wheel \
    && pip install "apache-airflow[statsd]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}" \
    && pip install pyOpenSSL \
    && pip install ffmpeg-python

It results in the following error:

  Downloading Flask_SQLAlchemy-2.4.4-py2.py3-none-any.whl (17 kB)
Collecting Flask-OpenID<2,>=1.2.5
  Downloading Flask-OpenID-1.2.5.tar.gz (43 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python3.7 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-m76ljhex/flask-openid_ea4f1905c75b4a29824cae4d51521afd/setup.py'"'"'; __file__='"'"'/tmp/pip-install-m76ljhex/flask-openid_ea4f1905c75b4a29824cae4d51521afd/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-ccsz9_u0
         cwd: /tmp/pip-install-m76ljhex/flask-openid_ea4f1905c75b4a29824cae4d51521afd/
    Complete output (1 lines):
    error in Flask-OpenID setup command: use_2to3 is invalid.
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/d1/a2/9d1fba3287a65f81b9d1c09c4f7cb16f8ea4988b1bc97ffea0d60983338f/Flask-OpenID-1.2.5.tar.gz#sha256=5a8ffe1c8c0ad1cc1f5030e1223ea27f8861ee0215a2a58a528cc61379e5ccab (from https://pypi.org/simple/flask-openid/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
INFO: pip is looking at multiple versions of flask to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of dill to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cffi to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cryptography to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of croniter to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of click to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of clickclick to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of connexion[flask,swagger-ui] to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of commonmark to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of colorlog to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of colorama to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cattrs to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cached-property to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of argcomplete to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of alembic to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of rich to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of apache-airflow[statsd] to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    flask-appbuilder 3.1.1 depends on Flask-OpenID<2 and >=1.2.5
    The user requested (constraint) flask-openid==1.2.5

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
The command '/bin/sh -c pip3 install --upgrade pip &&     pip install -U pip setuptools wheel     && pip install "apache-airflow[statsd]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"     && pip install pyOpenSSL     && pip install ffmpeg-python' returned a non-zero code: 1

potiuk · 2021-09-08T11:30:36Z

2.0.1 is not yet fixed (Only 2.1.3). I can fix it in 2.0.1 for your tests now

potiuk · 2021-09-08T11:37:08Z

try now @RafayAK - it should be ok for 2.0.1 as well

potiuk · 2021-09-08T11:38:50Z

(I need to manually apply the fix for all versions, so I want to confirm it works :) )

RafayAK · 2021-09-08T11:56:24Z

@potiuk Got it built thanks!

I do get this warning though:

Collecting certifi>=2017.4.17
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
WARNING: The candidate selected for download or install is a yanked version: 'sqlalchemy-utils' candidate (version 0.36.8 at https://files.pythonhosted.org/packages/14/68/e5301c4c960c79a32333b8805e52cb69d3d237aa869a773b4157ccb3eb26/SQLAlchemy-Utils-0.36.8.tar.gz#sha256=fb66e9956e41340011b70b80f898fde6064ec1817af77199ee21ace71d7d6ab0 (from https://pypi.org/simple/sqlalchemy-utils/))
Reason for being yanked: Wrong required python

potiuk · 2021-09-08T11:58:59Z

No problem with that warning - it's just yanked because of wrong setup information, but that's not an issue.

Thanks for verification!

I proceed with applying the fix then :)

potiuk · 2021-09-08T12:20:19Z

All right. Patched all released constraints.

For the record and keeping track:

2.1.3: 64508c7
2.1.2: 641d6ca
2.1.1: cf0834b
2.1.0: 1e52c69
2.0.2: 82cc57b
2.0.1: 2984d41
2.0.0: dd3e88f

I will close the issue where we also solve it in the "main" and apply to upcoming 2.1.4 (we still need to automate that as it was manual)

venkaa6 · 2021-09-08T13:03:24Z

hey team I am using airflow 1.10.10 and just faced this error on build can we apply to lower versions as well? @potiuk 🙏

uranusjr · 2021-09-08T13:14:35Z

You can apply this line to your requirements yourself:

Flask-OpenID @ https://github.com/apache/airflow-flask-openid-fork/releases/download/v1.2.6/Flask-OpenID-1.2.6.tar.gz

A recent pip version should pick up the line automatically. If you’re stuck on an old pip version, pip install the line separately before installing Airflow.

potiuk · 2021-09-08T13:20:52Z

Yep. With 1.10 you are pretty much on your own. 1.10 reached end-of-life and we are not going to make any workaronds or even critical security fixes there. As @uranusjr wrote - you have to workaround it yourself with the patch, or (might be simpler) run pip install setuptools==58.0.1 before installing Airflow 1.10.

venkaa6 · 2021-09-08T13:26:26Z

ah cool. yes thanks @uranusjr that worked!

See #18075

potiuk · 2021-09-08T14:33:43Z

BTW. I am also discussing with other people affected and we tapped into our network to possibly fix the problem by releasing updated Flask-OpenID, In which case we will be able to fix it "permanently" and not as workaround.

🤞

agrounds · 2021-09-08T15:02:11Z

@potiuk Airflow installation via pip is now failing for us. We're running this:

pip install -r requirements.txt --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

with AIRFLOW_VERSION=2.1.3 and PYTHON_VERSION=3.8, and we're getting this error:

DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement.. You can find discussion regarding this at pypa/pip#8210.
ERROR: Links are not allowed as constraints

pip version is 21.0.1, can also provide requirements.txt if it's helpful, but I don't think it's relevant.

More context: pallets-eco/flask-openid#60 apache/airflow#18075 Using Airflow patched fork for now

potiuk · 2021-09-08T15:49:52Z

pip version is 21.0.1, can also provide requirements.txt if it's helpful, but I don't think it's relevant.

You should upgrade to newer PIP version.

pip install --upgrade pip will do

As of 6th of September 2021, setuptools released 50.0.2 followed by 50.0.3 release - both of which fail Flask-OpenId installation. This broken Airflow installation. It's been workarounded for released version of Airflow by publishing patched version of FlaskOpenID in Airflow GitHub repository. This change adapts our main and constraint generation to include automated generation of the special Flask-OpenId dependency. With this change: 1) Setuptools is pinned to 50.0.3 (to make sure our builds work with that version - this can also give us more control in the future. 2) Flask-OpenID is installed explicitly before airflow is installed when eager-upgrade is used Closes: apache#18075

potiuk · 2021-09-08T22:29:28Z

This has been fixed now. Thanks to @mitsuhiko I became maintainer of the Flask-OpenID and I released new 1.3.0 version, Python3 - only with removed 2to3 configuration. It should nicely install now with the new setuptools !

potiuk · 2021-09-08T22:30:14Z

https://pypi.org/project/Flask-OpenID/1.3.0/

graingert · 2021-09-09T07:53:19Z

This has been fixed now. Thanks to @mitsuhiko I became maintainer of the Flask-OpenID and I released new 1.3.0 version, Python3 - only with removed 2to3 configuration. It should nicely install now with the new setuptools !

thanks, can confirm:

pip install flask-openid
Collecting flask-openid
  Downloading Flask_OpenID-1.3.0-py3-none-any.whl (9.3 kB)
Collecting python3-openid>=2.0
  Downloading python3_openid-3.2.0-py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 1.1 MB/s 
Collecting Flask>=0.10.1
  Using cached Flask-2.0.1-py3-none-any.whl (94 kB)

and there's a wheel so no setuptools needed!

potiuk · 2021-09-09T08:26:09Z

Yeah. I also released wheel on request of @uranusjr and @kaxil :).

Glad we could solve it that quickly.

I am going to delete https://github.com/apache/airflow-flask-openid-fork today.

apache/airflow#18075

edgarrmondragon added area:core kind:bug This is a clearly a bug labels Sep 8, 2021

potiuk added a commit that referenced this issue Sep 8, 2021

Switch main to patched version of Flask OpenId

520291e

See #18075

potiuk mentioned this issue Sep 8, 2021

Remove depreacted use2to3 option from setup.py pallets-eco/flask-openid#60

Merged

dpgaspar added a commit to dpgaspar/Flask-AppBuilder that referenced this issue Sep 8, 2021

fix: Flask-OpenID dependency, install breaks

539c66d

More context: pallets-eco/flask-openid#60 apache/airflow#18075 Using Airflow patched fork for now

dpgaspar mentioned this issue Sep 8, 2021

fix: Flask-OpenID dependency, install breaks dpgaspar/Flask-AppBuilder#1690

Closed

6 tasks

potiuk mentioned this issue Sep 8, 2021

Possible PyJWT dependency upgrade? #18091

Closed

2 tasks

potiuk mentioned this issue Sep 8, 2021

Workaround Flask-OpenId failure with recent setuptools update #18093

Closed

potiuk closed this as completed Sep 8, 2021

jaraco mentioned this issue Sep 12, 2021

[BUG] Should fail hard when installing packages that want 2to3 pypa/setuptools#2769

Closed

1 task

adlersantos mentioned this issue Sep 20, 2021

fix: Use patched flask-openid version to fix failing builds GoogleCloudPlatform/public-datasets-pipelines#188

Merged

2 tasks

fridex mentioned this issue Sep 22, 2021

Add prescription for installing flask-openid with the right setuptools thoth-station/prescriptions#18504

Merged

Thingus added a commit to Flowminder/FlowKit that referenced this issue Oct 6, 2021

Adding workaround for Airflow build

b1173ce

apache/airflow#18075

potiuk mentioned this issue Oct 21, 2022

Install without resolution on packages with pre-locked requirements pypa/pip#11528

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Airflow installation fails when with latest `setuptools` #18075

Airflow installation fails when with latest `setuptools` #18075

edgarrmondragon commented Sep 8, 2021 •

edited

Loading

boring-cyborg bot commented Sep 8, 2021

uranusjr commented Sep 8, 2021

potiuk commented Sep 8, 2021 •

edited

Loading

potiuk commented Sep 8, 2021

potiuk commented Sep 8, 2021

RafayAK commented Sep 8, 2021

RafayAK commented Sep 8, 2021

potiuk commented Sep 8, 2021

potiuk commented Sep 8, 2021

RafayAK commented Sep 8, 2021

potiuk commented Sep 8, 2021

RafayAK commented Sep 8, 2021

potiuk commented Sep 8, 2021 •

edited

Loading

potiuk commented Sep 8, 2021

potiuk commented Sep 8, 2021

RafayAK commented Sep 8, 2021

potiuk commented Sep 8, 2021

potiuk commented Sep 8, 2021

venkaa6 commented Sep 8, 2021

uranusjr commented Sep 8, 2021

potiuk commented Sep 8, 2021 •

edited

Loading

venkaa6 commented Sep 8, 2021

potiuk commented Sep 8, 2021

agrounds commented Sep 8, 2021 •

edited

Loading

potiuk commented Sep 8, 2021 •

edited

Loading

potiuk commented Sep 8, 2021

potiuk commented Sep 8, 2021

graingert commented Sep 9, 2021

potiuk commented Sep 9, 2021

Airflow installation fails when with latest setuptools #18075

Airflow installation fails when with latest setuptools #18075

Comments

edgarrmondragon commented Sep 8, 2021 • edited Loading

Apache Airflow version

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

What happened

What you expected to happen

How to reproduce

Anything else

Are you willing to submit PR?

Code of Conduct

boring-cyborg bot commented Sep 8, 2021

uranusjr commented Sep 8, 2021

potiuk commented Sep 8, 2021 • edited Loading

potiuk commented Sep 8, 2021

potiuk commented Sep 8, 2021

RafayAK commented Sep 8, 2021

RafayAK commented Sep 8, 2021

potiuk commented Sep 8, 2021

potiuk commented Sep 8, 2021

RafayAK commented Sep 8, 2021

potiuk commented Sep 8, 2021

RafayAK commented Sep 8, 2021

potiuk commented Sep 8, 2021 • edited Loading

potiuk commented Sep 8, 2021

potiuk commented Sep 8, 2021

RafayAK commented Sep 8, 2021

potiuk commented Sep 8, 2021

potiuk commented Sep 8, 2021

venkaa6 commented Sep 8, 2021

uranusjr commented Sep 8, 2021

potiuk commented Sep 8, 2021 • edited Loading

venkaa6 commented Sep 8, 2021

potiuk commented Sep 8, 2021

agrounds commented Sep 8, 2021 • edited Loading

potiuk commented Sep 8, 2021 • edited Loading

potiuk commented Sep 8, 2021

potiuk commented Sep 8, 2021

graingert commented Sep 9, 2021

potiuk commented Sep 9, 2021

Airflow installation fails when with latest `setuptools` #18075

Airflow installation fails when with latest `setuptools` #18075

edgarrmondragon commented Sep 8, 2021 •

edited

Loading

potiuk commented Sep 8, 2021 •

edited

Loading

potiuk commented Sep 8, 2021 •

edited

Loading

potiuk commented Sep 8, 2021 •

edited

Loading

agrounds commented Sep 8, 2021 •

edited

Loading

potiuk commented Sep 8, 2021 •

edited

Loading