Skip to content

Commit

Permalink
Slightly improve description of Dynamic DAG generation preamble (#28650)
Browse files Browse the repository at this point in the history
The description is more clear now what Dynamic DAG generation is
vs. Dynamic Task Mapping and note is added to the users to pay
attention about the stable sorting that should be applied when
generating DAGS.

Related: #27523
  • Loading branch information
potiuk authored Dec 30, 2022
1 parent c22fc00 commit 36d887b
Showing 1 changed file with 12 additions and 1 deletion.
13 changes: 12 additions & 1 deletion docs/apache-airflow/howto/dynamic-dag-generation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,18 @@
Dynamic DAG Generation
======================

To have a task repeated based on the output/result of a previous task see :doc:`/authoring-and-scheduling/dynamic-task-mapping`.
This document describes creation of DAGs that have a structure generated dynamically, but where the number of
tasks in the DAG does not change between DAG Runs. If you want to implement a DAG where number of Tasks (or
Task Groups as of Airflow 2.6) can change based on the output/result of previous tasks, see
:doc:`/authoring-and-scheduling/dynamic-task-mapping`.

.. note:: Consistent sequence of generating tasks and task groups

In all cases where you generate DAGs dynamically, you should make sure that Tasks and Task Groups
are generated with consistent sequence every time the DAG is generated, otherwise you might end up with
Tasks and Task Groups changing their sequence in the Grid View every time you refresh the page.
This can be achieved for example by using a stable sorting mechanism in your Database queries or by using
``sorted()`` function in Python.

Dynamic DAGs with environment variables
.......................................
Expand Down

0 comments on commit 36d887b

Please sign in to comment.