Fix #2735 - Simplify Airflow properties extraction #2749

pmbrull · 2022-02-14T11:42:00Z

Describe your changes :

This PR fixes #2735

We were encountering a known issue (apache/airflow#20875) when trying to extract DAG and Task properties using airflow's serialiser. That issue was fixed after 2.2.3 release (so we would still need to wait for 2.2.4).

In this PR we are ditching the serialiser completely and instead, we use Airflow models for DAG, DagRun and TaskInstance to extract all necessary data. As all this information is available in the context of the lineage and callbacks, we can just pick this info up from the actual Airflow objects.

This lets us play safely on 2.2.3. However, for Airflow 1.10.15, further work is required. More info #2748

Thanks!

Type of change :

Frontend Preview (Screenshots) :

For frontend related change, please link screenshots of your changes preview! Optional for backend related changes.

Checklist:

I have read the CONTRIBUTING document.
I have performed a self-review of my own.
I have tagged my reviewers below.
I have commented on my code, particularly in hard-to-understand areas.
My changes generate no new warnings.
I have added tests that prove my fix is effective or that my feature works.
All new and existing tests passed.

Reviewers

pmbrull · 2022-02-14T11:42:56Z

docker/local-metadata/docker-compose.yml

@@ -91,6 +91,7 @@ services:

 networks:
  local_app_net:
+    name: ometa_network


This is helpful when hooking up different docker-compose runs. I needed it when trying to align the services between a compose running OM and another one running Airflow 1.10.15

github-actions · 2022-02-14T11:43:44Z

The Python checkstyle failed.

Please run make py_format in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.