Add cli command for 'airflow dags reserialize`

[collinmcnulty]

Dag serialization is currently out of the hands of the user. Whenever dag reserialization is required, I run this python script:

from airflow.models.serialized_dag import SerializedDagModel
from airflow.settings import Session
session = Session()
session.query(SerializedDagModel).delete()
session.commit()

This PR makes running that script as simple as airflow dags reserialize.

closes: #19432

[potiuk]

Doc build is failing. I think it would also be good to add this info to the troubleshooting section in the upcoming : upgrading page: #19453

[kaxil]

I think this needs to be thought through.

1. It should be nested under airflow dags reserialize
2. For actual reserialize we should serialize after deleting too rather than waiting for dag parsing process I think
3. Needs some unit tests

[collinmcnulty]

1. I thought I did nest it there? That was my intention. Am I misreading something?
2. I was thinking that it would be better to keep all the dag parsing logs together, but that's not a very strongly held opinion.
3. Will do.

[kaxil]

(2) As the name is reserialize -- it will only reserialize if the scheduler is running. If the scheduler is not running for whatever reason, the users will have a wrong impression that reserialize does actually deserialize instead of just "clear"/delete

[dstandish]

@collinmcnulty can you share some context re under what circumstances you find that this needs to be done? perhaps there is a change we should make to the dag parsing process that makes it so we don't need to jump in there manually

[collinmcnulty]

@dstandish The biggest was part of the 2.2 upgrade where the Param change was supposed to be backwards compatible, but wasn't strictly. I've also seen cases where a Dag Import error didn't show up because there was a valid but outdated, serialized version of the dag. Not sure if the latter case is still present in main. I can check.

[collinmcnulty]

Docs are failing because they do not like the word "reserialize".

[dstandish]

there's a spelling_wordlist.txt you should be able to modify

[dstandish]

The biggest was part of the 2.2 upgrade where the Param change was supposed to be backwards compatible, but wasn't strictly. I've also seen cases where a Dag Import error didn't show up because there was a valid but outdated, serialized version of the dag. Not sure if the latter case is still present in main. I can check.

interesting i wonder if perhaps it would be good to have it automatically reserialize at the end of an upgrade. i'm not sure with what frequency this is done currently, but i would have assumed it would be done with each parsing interval like every 5 minutes or something 🤔

[collinmcnulty]

@dstandish It will reserialize if it successfully parses. But if you break a DAG, then it will keep the old serialized version. See more discussion over longer term fixes in #19367

[dstandish]

@dstandish It will reserialize if it successfully parses. But if you break a DAG, then it will keep the old serialized version. See more discussion over longer term fixes in #19367

thanks, will take a look over there. but if the dag is broken, reserializing wouldn't work anyway would it?

[collinmcnulty]

It won't "work" but it will surface the Import error and will stop trying to schedule a broken dag

[kaxil]

#19367 / #18120 (WIP) will hopefully fix the underlying issue

But a CLI command to show serialized representation, actually serialize from a dag file or clear -- all would be very handy

[potiuk]

#19367 / #18120 (WIP) will hopefully fix the underlying issue

But a CLI command to show serialized representation, actually serialize from a dag file or clear -- all would be very handy

Agree. That's a good idea to add more tooling here - not only reserializing but outputing serialized dags nicely formatted and coloured using pygments - that woudl be reallly nice.

[potiuk]

will you rebase/continue that @collinmcnulty ?

[collinmcnulty]

Yes I have rebased (back from vacation). I was not planning on adding any more features to this PR. Adding the display of the serialized dags with pygments is something I would have to study how to do. I was thinking it might be worthwhile to review this PR as is and open a separate issue to add the additional tooling. If you disagree, I can look at adding this tooling, but it will take me a little longer.

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[EricGao888]

Our team are using airflow with Celery Executor. For example, we have a cluster of 1 header machine and 2 worker machines. We deploy webserver and scheduler on header and have the workers run on worker machines. If I run airflow dags reserialize on header machine, will the two workers get updated? Will there be any consistent issues? Should I run airflow dags reserialize on both header and workers? Thanks.

[uranusjr]

Workers don’t need DAG information, only the scheduler does.

[EricGao888]

Workers don’t need DAG information, only the scheduler does.

Thanks for the reply. Another question is will the python script from airflow.models.serialized_dag import SerializedDagModel from airflow.settings import Session session = Session() session.query(SerializedDagModel).delete() session.commit() work for airflow 1.10.12?

I found the some info from https://airflow.apache.org/docs/apache-airflow/stable/dag-serialization.html
It seems in airflow 1.10.12, the scheduler does not use Serialized DAGS. Does it mean the sync py script in #19471 does not work for airflow 1.10.12? I did a quick test and didn’t get the dags synced in airflow 1.10.12. Did I miss any important information?

[potiuk]

Did I miss any important information?

I think you missed the information that Airlfow 1.10 (not even 1.10.12 but any 1.10 version) reached end of life in June 2021 and will not receive any updates (not even critical fixes) any more. We do not produce any code that is targetted for 1.10 for about half a year already.

See here for example: https://github.com/apache/airflow#version-life-cycle

Please migrate to Airflow 2 as soon as possible

[EricGao888]

Did I miss any important information?

I think you missed the information that Airlfow 1.10 (not even 1.10.12 but any 1.10 version) reached end of life in June 2021 and will not receive any updates (not even critical fixes) any more. We do not produce any code that is targetted for 1.10 for about half a year already.

See here for example: https://github.com/apache/airflow#version-life-cycle

Please migrate to Airflow 2 as soon as possible

Thanks for the reply. Will migrate to Airflow 2 ASAP.