Google provider package should have google-ads<18.1.0 constraint

[seongwon77]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

alembic==1.8.1
anyio==3.6.1
apache-airflow==2.3.4
apache-airflow-providers-common-sql==1.1.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-sqlite==3.2.0
apispec==3.3.2
argcomplete==2.0.0
attrs==22.1.0
Babel==2.10.3
blinker==1.5
cachelib==0.9.0
cachetools==5.2.0
cattrs==22.1.0
certifi==2022.6.15
cffi==1.15.1
charset-normalizer==2.1.1
click==8.1.3
clickclick==20.10.2
colorama==0.4.5
colorlog==4.8.0
commonmark==0.9.1
connexion==2.14.1
cron-descriptor==1.2.31
croniter==1.3.5
cryptography==37.0.4
db-dtypes==1.0.3
Deprecated==1.2.13
dill==0.3.5.1
dnspython==2.2.1
docutils==0.19
email-validator==1.2.1
exceptiongroup==1.0.0rc8
Flask==2.2.2
Flask-AppBuilder==4.1.3
Flask-Babel==2.0.0
Flask-Caching==2.0.1
Flask-JWT-Extended==4.4.4
Flask-Login==0.6.2
Flask-Session==0.4.0
Flask-SQLAlchemy==2.5.1
Flask-WTF==0.15.1
future==0.18.2
google-ads==18.1.0
google-api-core==2.8.2
google-api-python-client==1.12.11
google-auth==2.11.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.5.2
google-cloud-aiplatform==1.13.1
google-cloud-appengine-logging==1.1.4
google-cloud-audit-log==0.2.4
google-cloud-automl==2.8.1
google-cloud-bigquery==2.34.3
google-cloud-bigquery-datatransfer==3.7.1
google-cloud-bigquery-storage==2.14.2
google-cloud-bigtable==1.7.1
google-cloud-build==3.9.1
google-cloud-container==2.11.2
google-cloud-core==2.3.2
google-cloud-datacatalog==3.9.1
google-cloud-dataform==0.2.1
google-cloud-dataplex==1.1.1
google-cloud-dataproc==5.0.1
google-cloud-dataproc-metastore==1.6.1
google-cloud-dlp==1.0.1
google-cloud-kms==2.12.1
google-cloud-language==1.3.1
google-cloud-logging==3.2.2
google-cloud-memcache==1.4.2
google-cloud-monitoring==2.11.1
google-cloud-orchestration-airflow==1.4.2
google-cloud-os-login==2.7.2
google-cloud-pubsub==2.13.6
google-cloud-redis==2.9.1
google-cloud-resource-manager==1.6.1
google-cloud-secret-manager==1.0.1
google-cloud-spanner==1.19.2
google-cloud-speech==1.3.3
google-cloud-storage==1.44.0
google-cloud-tasks==2.10.2
google-cloud-texttospeech==1.0.2
google-cloud-translate==1.7.1
google-cloud-videointelligence==1.16.2
google-cloud-vision==1.0.1
google-cloud-workflows==1.7.2
google-crc32c==1.3.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.4
graphviz==0.20.1
grpc-google-iam-v1==0.12.4
grpcio==1.47.0
grpcio-gcp==0.2.2
grpcio-status==1.47.0
gunicorn==20.1.0
h11==0.12.0
httpcore==0.15.0
httplib2==0.20.4
httpx==0.23.0
idna==3.3
importlib-metadata==4.12.0
importlib-resources==5.9.0
inflection==0.5.1
itsdangerous==2.1.2
Jinja2==3.1.2
json-merge-patch==0.2
jsonschema==4.14.0
lazy-object-proxy==1.7.1
linkify-it-py==2.0.0
lockfile==0.12.2
looker-sdk==22.10.0
Mako==1.2.1
Markdown==3.4.1
markdown-it-py==2.1.0
MarkupSafe==2.1.1
marshmallow==3.17.1
marshmallow-enum==1.5.1
marshmallow-oneofschema==3.0.1
marshmallow-sqlalchemy==0.26.1
mdit-py-plugins==0.3.0
mdurl==0.1.2
numpy==1.23.2
oauthlib==3.2.0
packaging==21.3
pandas==1.4.3
pandas-gbq==0.17.8
pathspec==0.9.0
pendulum==2.1.2
pkgutil_resolve_name==1.3.10
pluggy==1.0.0
prison==0.2.1
proto-plus==1.22.0
protobuf==4.21.5
psutil==5.9.1
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pydata-google-auth==1.4.0
Pygments==2.13.0
PyJWT==2.4.0
pyOpenSSL==22.0.0
pyparsing==3.0.9
pyrsistent==0.18.1
python-daemon==2.3.1
python-dateutil==2.8.2
python-nvd3==0.15.0
python-slugify==6.1.2
pytz==2022.2.1
pytzdata==2020.1
PyYAML==6.0
requests==2.28.1
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
rfc3986==1.5.0
rich==12.5.1
rsa==4.9
setproctitle==1.3.2
six==1.16.0
sniffio==1.2.0
SQLAlchemy==1.4.27
sqlalchemy-bigquery==1.4.4
SQLAlchemy-JSONField==1.0.0
SQLAlchemy-Utils==0.38.3
sqlparse==0.4.2
swagger-ui-bundle==0.0.9
tabulate==0.8.10
tenacity==8.0.1
termcolor==1.1.0
text-unidecode==1.3
typing_extensions==4.3.0
uc-micro-py==1.0.1
unicodecsv==0.14.1
uritemplate==3.0.1
urllib3==1.26.12
Werkzeug==2.2.2
wrapt==1.14.1
WTForms==2.3.3
zipp==3.8.1

Apache Airflow version

2.3.4

Operating System

Debian 11 bullseye

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

What happened

Starting from 18.1.0 of the package google-ads, one of its constraint has change, which is protobuf >= 4.21.5.

• Before: https://github.com/googleads/google-ads-python/blob/18.0.0/setup.py#L34
• After: https://github.com/googleads/google-ads-python/blob/18.1.0/setup.py#L31

With 18.1.0 of google-ads, it tries to install protobuf==4.21.5. Then, the Airflow DB migration job fails when initiating an Airflow instance with the following log. (I deployed the instance to a k8s cluster using the official Helm chart)

ERROR! Maximum number of retries (20) reached.

{}

Last check result:

$ airflow db check

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__init__.py", line 35, in <module>
    from airflow import settings
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py", line 35, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py", line 1618, in <module>
    secrets_backend_list = initialize_secrets_backends()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py", line 1540, in initialize_secrets_backends
    custom_secret_backend = get_custom_secret_backend()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py", line 1515, in get_custom_secret_backend
    secrets_backend_cls = conf.getimport(section='secrets', key='backend')
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py", line 684, in getimport
    return import_string(full_qualified_path)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
    module = import_module(module_path)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/secrets/secret_manager.py", line 27, in <module>
    from airflow.providers.google.cloud._internal_client.secret_manager_client import _SecretManagerClient
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/_internal_client/secret_manager_client.py", line 23, in <module>
    from google.cloud.secretmanager_v1 import SecretManagerServiceClient
  File "/home/airflow/.local/lib/python3.8/site-packages/google/cloud/secretmanager_v1/__init__.py", line 22, in <module>
    from google.cloud.secretmanager_v1 import types
  File "/home/airflow/.local/lib/python3.8/site-packages/google/cloud/secretmanager_v1/types.py", line 23, in <module>
    from google.cloud.secretmanager_v1.proto import resources_pb2
  File "/home/airflow/.local/lib/python3.8/site-packages/google/cloud/secretmanager_v1/proto/resources_pb2.py", line 57, in <module>
    _descriptor.EnumValueDescriptor(
  File "/home/airflow/.local/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 755, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.

If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.

If you cannot immediately regenerate your protos, some other possible workarounds are:

 1. Downgrade the protobuf package to 3.20.x or lower.

 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

{}

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

What you think should happen instead

Looks like the version of the package must be equal to or lower than 3.20.x.

How to reproduce

1. Build a custom Airflow image with the following Dockerfile.

FROM apache/airflow:slim-2.3.4-python3.8
RUN pip install apache-airflow[google]

2. Add a configuration to set the secret backend to Google Secret Manager.

3. Deploy a new Airflow instance to a k8s cluster using the official Helm chart.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Thanks for notifying us.

It's already been handled few days ago (our CI tests caught the problem). We handled that by limiting the protobuf rather than ads. #25886. It will be released in the next version.

BTW, It's not 'constraint' - it's requirement :). Those two have completely different meaning in pip (and we are using constraints heavily so it makes a difference)

[seongwon77]

Oh, thanks for letting me know. Glad to see the next release! :)

[potiuk]

BTW. I am also talking to Google team - they need to upgrade the integraions soon for some old clients as otherwise things like that will start happening more and more often