Support Airflow 1.10.15

[pmbrull]

Here we can find the constraints necessary to run airflow 1.10.15. (from airflow installation guide)

If we want to run our backend lineage on Airflow, openmetadata-ingestion needs to be installed to the Airflow instances, but currently, there are some dependencies mismatching, e.g.:

| airflow | openmetadata |
| --- | --- |
| requests==2.23.0 | requests~=2.26 |
| google-auth==1.23.0 | google-auth>=1.33.0 |
| click==6.7 | click>=7.1.1 |
...

This is the error message I got trying to get everything running on 1.10.15:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-cloud-storage 1.33.0 requires google-auth<2.0dev,>=1.11.0, but you have google-auth 2.6.0 which is incompatible.
google-api-core 1.23.0 requires google-auth<2.0dev,>=1.21.1, but you have google-auth 2.6.0 which is incompatible.
flask-appbuilder 2.3.4 requires click<8,>=6.7, but you have click 8.0.3 which is incompatible.
apache-airflow 1.10.15 requires requests<2.24.0,>=2.20.0; python_version >= "3.0", but you have requests 2.27.1 which is incompatible.

In order to be able to support Airflow 1 I'd propose to break down our current openmetadata-ingestion package into smaller pieces:

• Python API
• Airflow Backend
• Ingestion Framework <- here we have the heavy dependencies, but they do not need to live in the Airflow host

After this setup, we should be able to prepare a more flexible approach on our Airflow Backend and properly test on 1.10.15

Thanks!

cc @harshach

[harshach]

@pmbrull I think we have multiple tickets around this issue