2.2.4+twtr #81
2.2.4+twtr #81
Conversation
…ob scheduling without explicit_defaults_for_timestamp (twitter-forks#6) * fb64f2e: [TWTR][AIRFLOW-XXX] Twitter Airflow Customizations + Fixup job scheduling without explicit_defaults_for_timestamp * reformat * flake8 fix
…ter-forks#13) * [EWT-16]: Airflow fix for manual trigger during version upgrade
* 99ee040: CP from 1.10+twtr * 2e01c24: CP from 1.10.4 ([TWTR][AIRFLOW-4939] Fixup use of fallback kwarg in conf.getint) * 00cb4ae: [TWTR][AIRFLOW-XXXX] Cherry-pick d4a83bc and bump version (twitter-forks#21) * CP 51b1aee: Relax version requiremets (twitter-forks#24) * CP 67a4d1c: [CX-16266] Change with reference to 1a4c164 commit in open source (twitter-forks#25) * CP 54bd095: [TWTR][CX-17516] Queue tasks already being handled by the executor (twitter-forks#26) * CP 87fcc1c: [TWTR][CX-17516] Requeue tasks in the queued state (twitter-forks#27) * CP 98a1ca9: [AIRFLOW-6625] Explicitly log using utf-8 encoding (apache#7247) (twitter-forks#31) * fixing models.py and jobs.py file fix after CP * fixing typo and version bump Co-authored-by: Vishesh Jain <[email protected]>
…scheduler_ddavydov (twitter-forks#42)
…leloc (missed in EWT-16) (twitter-forks#39) Co-authored-by: Vishesh Jain <[email protected]> Co-authored-by: aoen <[email protected]>
| @@ -0,0 +1,6 @@ | |||
| { | |||
There was a problem hiding this comment.
Is this file supposed to be merged?
There was a problem hiding this comment.
This change got added by the following CP fb64f2ee590ccfd8656dfc163ff2bed41e821571.
| @@ -0,0 +1,20 @@ | |||
| # Developing locally | |||
There was a problem hiding this comment.
Are these steps valid and correct?
There was a problem hiding this comment.
Steps 1 -8, are validated, let me validate and confirm step 9. Though the changes are picked from a CP, I have made a change in step 2 (change version.py to setup.py) and step 9. We can also refine this file (later after CP) to add uploading the wheels to staging artifactory and using it in the local machine
There was a problem hiding this comment.
Can you verify the steps by trying running them?
| @@ -0,0 +1,217 @@ | |||
| # Licensed to the Apache Software Foundation (ASF) under one | |||
There was a problem hiding this comment.
I think this file should not be here. This should have been refactored into something else.
I do not remember, we have added all this logic.
| @@ -0,0 +1,371 @@ | |||
| # Licensed to the Apache Software Foundation (ASF) under one | |||
There was a problem hiding this comment.
I do not remember we added this file or logic.
| ) | ||
| env['AIRFLOW__CORE__DAGS_FOLDER'] = dag_volume_mount_path | ||
|
|
||
| # TODO This change can be submitted into the apache as well. |
| try_number, airflow_command, kube_executor_config): | ||
| volumes_dict, volume_mounts_dict = self._get_volumes_and_mounts() | ||
| worker_init_container_spec = self._get_init_containers() | ||
| resources = Resources( |
There was a problem hiding this comment.
We may have added this logic or something around configuring the resources for worker task pods.
| @@ -785,10 +785,17 @@ def _do_scheduling(self, session) -> int: | |||
|
|
|||
| By "next oldest", we mean hasn't been examined/scheduled in the most time. | |||
|
|
|||
| <<<<<<< HEAD | |||
| @@ -0,0 +1,2633 @@ | |||
| # -*- coding: utf-8 -*- | |||
There was a problem hiding this comment.
I am not sure if we have added all this logic. @vshshjn7 can you please check and confirm?
Airflow Upgrade to 2.2.4 with Cherry-Picks [only] to apply twitter specific patches with some minor modifications
Cherry Picked Commits:
76fe7ac -> Airflow Customizations + Fixup job scheduling without explicit_defaults_for_timestamp , reformat and flake8 fix.
91d2b00 -> Airflow fix for manual trigger during version upgrade.
Note: The test files were left out during CP
4ce8d4c -> Fix for rendering code on UI
299b4d8 -> [TWTR] CP from 1.10+twtr ([CP from 1.10+twtr #35]
8a689af -> Export scheduler env variable into worker pods.
e9642c2 -> Fetch task logs from worker pods. The change was already present in the repo
5875a15 -> [EWT-115][EWT-118] Initialize dag var to None and fix for DagModel.fileloc (missed in EWT-16) ([EWT-115][EWT-118]dag = None and fix for DagModel.fileloc(missed in EWT-16) #39)