Status of testing of Apache Airflow 2.2.5rc3

[ephraimbuddy]

Body

We have a kind request for all the contributors to the latest Apache Airflow RC 2.2.5rc3.

Could you please help us to test the RC versions of Airflow?

Please let us know in the comment if the issue is addressed in the latest RC.

• Set X-Frame-Options header to DENY unless X_FRAME_ENABLED is set to true (#19491): @subkanthi @planoe-cloudera
  Linked issues:
  • x_frame_enabled logic broken in Airflow 2 (#17255)
• Fix: Do not render undefined graph edges (#19684): @bbovenzi
• Fix Tasks getting stuck in scheduled state (#19747): @vapiravfif @tanelk
  Linked issues:
  • Tasks get stuck in "scheduled" state and starved when dags with huge amount of tasks is scheduled (#19622)
• Rename to_delete to to_cancel in TriggerRunner (#20658): @dstandish
• Fix duplicate trigger creation race condition (#20699): @MatrixManAtYrService @dstandish
  Linked issues:
  • TriggerEvent fires, and then defers a second time (doesn't fire a second time though). (#18392)
  • Track completed triggers until deleted from database (#19546)
• Truncate stack trace to DAG user code for exceptions raised during execution (#20731): @malthe
• fix: Update custom connection field processing (#20883): @mike-mcdonald
  Linked issues:
  • Cannot edit custom fields on provider connections (#20839)
• Ensure clear_task_instances sets valid run state (#21116): @uranusjr @uplsh580
  Linked issues:
  • Running airflow dags backfill --reset-dagruns <dag_id> -s <execution_start_dt> -e <execution_end_dt> results in error when run twice. (#21058)
  • Fix 'airflow dags backfill --reset-dagruns' errors when run twice (#21062)
• Update ExternalTaskSensorLink to handle templated external_dag_id (#21192): @csp98 @josh-fell
  Linked issues:
  • Webserver "External DAG" button on ExternalTaskSensor not working when dag_id is templated (#21183)
• A trigger might use a connection; make sure we mask passwords (#21207): @malthe
• Log traceback in trigger excs (#21213): @malthe
• Fix trigger dag redirect from task instance log view (#21239): @dstandish
• Fix race condition between triggerer and scheduler (#21316): @malthe
• Reduce DB load incurred by Stale DAG deactivation (#21399): @SamWheating
• Fix max_active_runs=1 not scheduling runs when min_file_process_interval is high (#21413): @ephraimbuddy
• Fix Resources eq check (#21442): @pingzh
• Filter out default configs when overrides exist. (#21539): @xyu @ashb
  Linked issues:
  • PR #18772 breaks sql_alchemy_conn_cmd config (#20092)
  • Don't bake ENV and _cmd into tmp config for non-sudo (#18772)
• Fix logging JDBC SQL error when task fails (#21540): @stijn-meersman @hubert-pietron @jyotsa09
  Linked issues:
  • No SQL error shown when using the JDBC operator (#18482)
  • JDBC operator not logging errors (#16295)
• Dispose not used connection pool (#21565): @pingzh
• FIX: extends typing-extensions to be installed with python 3.8+ #21566 (#21567): @frankcash
  Linked issues:
  • typing_extensions package isn't installed with apache-airflow-providers-amazon causing an issue for SqlToS3Operator (#21566)
• Disable default_pool delete on web ui (#21658): @chenglongyan
• Log exception in local executor (#21667): @uranusjr
• Correctly handle multiple '=' in LocalFileSystem secrets. (#21694): @kadai0308 @joeyberkovitz
  Linked issues:
  • LocalFileSystem Secret Backend Bug (#21625)
• Fix stray order_by(TaskInstance.execution_date) (#21705): @uranusjr @TidesMind
  Linked issues:
  • Airflow >= 2.2.0 execution date change is failing TaskInstance get_task_instances method and possibly others (#21656)
• Fix filesystem sensor for directories (#21729): @mixilchenko
• Fix graph autorefresh on page load (#21736): @bbovenzi
• Fix triggerer --capacity parameter (#21753): @thesuperzapper @msumit
  Linked issues:
  • triggerer --capacity parameter does not work (#21752)
• Fix assignment of unassigned triggers (#21770): @jkramer-ginkgo
• Fix incorrect data provided to tries & landing times charts (#21928): @millin
• Fixed dask executor and tests (#22027): @subkanthi @potiuk
  Linked issues:
  • Removes limitations from Dask dependencies (#22017)
• Add the new Airflow Trove Classifier to setup.cfg (#22241): @potiuk
• Fix broken links to celery documentation (#22364): @davidhagens
• Add back celery intersphinx mapping (#22370): @kaxil
• Replace timedelta.max with year long timdelta in test_manager (#22527): @potiuk
• Check and disallow a relative path for sqlite (#22530): @yenchenLiu

Thanks to all who contributed to the release (probably not a complete list!):
@frankcash @yenchenLiu @bbovenzi @SamWheating @ephraimbuddy @csp98 @joeyberkovitz @planoe-cloudera @hubert-pietron @dstandish @stijn-meersman @jyotsa09 @kadai0308 @chenglongyan @pingzh @TidesMind @josh-fell @potiuk @ashb @mike-mcdonald @vapiravfif @thesuperzapper @msumit @xyu @uplsh580 @tanelk @subkanthi @uranusjr @jkramer-ginkgo @millin @malthe @mixilchenko @davidhagens @MatrixManAtYrService @kaxil

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

All good.

Checked #22241 - > classifier is there

#21905 and #22527 were mostly to make sure the tests pass (they did :) )

#22027 - > I checked that dependencies are ok (below). Tests passed too. It would be great @subkanthi if you could check it (I have no Dask account ) :)

http https://pypi.org/pypi/apache-airflow/2.2.5rc1/json | jq ".info" | grep "extra == 'dask'":

    "cloudpickle (>=1.4.1) ; extra == 'dask'",
    "dask (>=2.9.0) ; extra == 'dask'",
    "distributed (>=2.11.1) ; extra == 'dask'",

[mixilchenko]

#21729 looks good for 2.2.5 release

[raphaelauv]

#18797 look good

with python 3.6

[2022-03-27, 18:45:42 UTC] {kubernetes_pod.py:525} INFO - Creating pod airflow-test-pod.2e925545978340a197ae86ee955e660a with labels: {'dag_id': 'kubernetes_dag', 'task_id': 'task-one', 'run_id': 'scheduled__2022-03-18T0000000000-90832bb28', 'try_number': '1'}
[2022-03-27, 18:45:42 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:43 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:44 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:45 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:47 UTC] {pod_manager.py:203} INFO - toto
[2022-03-27, 18:46:03 UTC] {pod_manager.py:203} INFO - finish
[2022-03-27, 18:46:04 UTC] {kubernetes_pod.py:418} INFO - Deleting pod: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:46:04 UTC] {taskinstance.py:1288} INFO - Marking task as SUCCESS. dag_id=kubernetes_dag, task_id=task-one, execution_date=20220318T000000, start_date=20220327T184542, end_date=20220327T184604
[2022-03-27, 18:46:04 UTC] {local_task_job.py:154} INFO - Task exited with return code 0
[2022-03-27, 18:46:04 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

with python 3.9

[2022-03-27, 18:29:26 UTC] {kubernetes_pod.py:525} INFO - Creating pod airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370 with labels: {'dag_id': 'kubernetes_dag', 'task_id': 'task-one', 'run_id': 'scheduled__2022-03-26T0000000000-09f4abc4d', 'try_number': '1'}
[2022-03-27, 18:29:27 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:28 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:29 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:30 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:31 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:32 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:33 UTC] {pod_manager.py:199} INFO - toto
[2022-03-27, 18:29:48 UTC] {pod_manager.py:199} INFO - finish
[2022-03-27, 18:29:49 UTC] {pod_manager.py:214} WARNING - Pod airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370 log read interrupted but container base still running
[2022-03-27, 18:29:50 UTC] {pod_manager.py:199} INFO - finish
[2022-03-27, 18:29:50 UTC] {kubernetes_pod.py:418} INFO - Deleting pod: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:50 UTC] {taskinstance.py:1278} INFO - Marking task as SUCCESS. dag_id=kubernetes_dag, task_id=task-one, execution_date=20220326T000000, start_date=20220327T182926, end_date=20220327T182950
[2022-03-27, 18:29:50 UTC] {local_task_job.py:154} INFO - Task exited with return code 0
[2022-03-27, 18:29:50 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

[joeyberkovitz]

#21694 looks good - tested with 2.2.5rc1 docker image.

conn.env:

TEST_CONN=mysql://username:password@host:1234/schema?param1=val1&param2=val2

airflow connections get TEST_CONN:

id   | conn_id   | conn_type | description | host | schema | login    | password | port | is_encrypted | is_extra_encrypted | extra_dejson                         | get_uri                                                           
=====+===========+===========+=============+======+========+==========+==========+======+==============+====================+======================================+===================================================================
None | TEST_CONN | mysql     | None        | host | schema | username | password | 1234 | False        | False              | {'param1': 'val1', 'param2': 'val2'} | mysql://username:password@host:1234/schema?param1=val1&param2=val2

[frankcash]

Just brought up a clean docker image with the RC to validate #21567

$ python --version && airflow version && pip list | grep typing_extension
Python 3.8.13
2.2.5rc1
typing_extensions                      4.1.1

Looks good to me

[subkanthi]

All good.

Checked #22241 - > classifier is there

#21905 and #22527 were mostly to make sure the tests pass (they did :) )

#22027 - > I checked that dependencies are ok (below). Tests passed too. It would be great @subkanthi if you could check it (I have no Dask account ) :)

http https://pypi.org/pypi/apache-airflow/2.2.5rc1/json | jq ".info" | grep "extra == 'dask'":

    "cloudpickle (>=1.4.1) ; extra == 'dask'",
    "dask (>=2.9.0) ; extra == 'dask'",
    "distributed (>=2.11.1) ; extra == 'dask'",

Tested #22027 on the tag 2.2.5rc1. Thanks

[mike-mcdonald]

Tested #20883 on 2.2.5rc2. Tests passed, using the connection form to test saving also worked as expected.

[jkramer-ginkgo]

#21770 <#21770> confirmed with heavy load

…

On Mon, Mar 28, 2022 at 12:06 PM Kanthi ***@***.***> wrote: All good. Checked #22241 <https://protect-us.mimecast.com/s/MBO2CkR0GkiZorph9lxCZ?domain=github.com> - > classifier is there [image: image] <https://protect-us.mimecast.com/s/T-b-C0Rr5NilYMLTwQ0Xm?domain=user-images.githubusercontent.com> #21905 <https://protect-us.mimecast.com/s/EAlGClY9JlUD6AvU1kMlB?domain=github.com> and #22527 <https://protect-us.mimecast.com/s/nf91CmZ9KmSGkArTDzQ7N?domain=github.com> were mostly to make sure the tests pass (they did :) ) #22027 <https://protect-us.mimecast.com/s/OejUCn59L0fQrKYIp_80p?domain=github.com> - > I checked that dependencies are ok (below). Tests passed too. It would be great @subkanthi <https://protect-us.mimecast.com/s/Xqz0Co2gMoUJ6B0c7JApt?domain=github.com> if you could check it (I have no Dask account ) :) http https://pypi.org/pypi/apache-airflow/2.2.5rc1/json <https://protect-us.mimecast.com/s/HRcMCgJ9AOfn4m7uo3y99?domain=pypi.org> | jq ".info" | grep "extra == 'dask'": "cloudpickle (>=1.4.1) ; extra == 'dask'", "dask (>=2.9.0) ; extra == 'dask'", "distributed (>=2.11.1) ; extra == 'dask'", Tested #22027 <https://protect-us.mimecast.com/s/OejUCn59L0fQrKYIp_80p?domain=github.com> on the tag 2.2.5rc1. Thanks — Reply to this email directly, view it on GitHub <https://protect-us.mimecast.com/s/Fb_bCjR9EjikZ3vT5RsRD?domain=github.com>, or unsubscribe <https://protect-us.mimecast.com/s/GaXnCpYjNpUPpvBUx8OSR?domain=github.com> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[frankcash]

Validated #21567

$ python --version && airflow version && pip list | grep typing_extension
Python 3.8.13
2.2.5rc2
typing_extensions                      4.1.1

[potiuk]

All good with RC3 from my point of view - including updated dependencies.

[josh-fell]

Verified #21192. Thanks @ephraimbuddy for organizing this release!

[eladkal]

Closing since 2.2.5 was released.