Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SPIKE: Move Macros to Task SDK #45440

Open
kaxil opened this issue Jan 6, 2025 · 4 comments
Open

SPIKE: Move Macros to Task SDK #45440

kaxil opened this issue Jan 6, 2025 · 4 comments
Labels
area:plugins area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK
Milestone
Airflow 3.0.0

Comments

@kaxil
Copy link
Member

kaxil commented Jan 6, 2025

Make a plan for porting https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#macros to Task SDK.

Macros are used in Jinja Templates and some of them are provided built-in and users can register some via Airflow Plugins: https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/plugins.html
@kaxil kaxil added the area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK label Jan 6, 2025
@dosubot dosubot bot added the area:plugins label Jan 6, 2025
@kaxil
Copy link
Member Author

kaxil commented Jan 7, 2025

Some of the macros that did not have new dependencies have been ported in #45444

@kaxil kaxil added this to the Airflow 3.0.0 milestone Jan 17, 2025
@vatsrahul1001
Copy link
Collaborator

@kaxil @amoghrajesh While testing our regression DAG's with airflow main we see our DAG's are failing as macros are not implemented. Do we plan to get this done before airflow alpha's?

More details here: https://astronomer.slack.com/archives/C01UJJEN0P3/p1738051755616179?thread_ts=1738051741.614759&cid=C01UJJEN0P3

@vatsrahul1001
Copy link
Collaborator

As discussed with @kaxil macros.ds_add and macros.random should work. I will verify this again.

kaxil added a commit to astronomer/airflow that referenced this issue Feb 18, 2025
@kaxil
Move Macros to Task SDK
1201d31 
Part of apache#45440

Built-in macros have been moved but the support to register a custom macro via plugin will be done as part of apache#45440.

This PR moves `random`, `dateutil`, `yaml`, `datetime_diff_for_humans`, `ds_format_locale` to Python Task SDK.

Backwards-compatibility has been maintained for now in `airflow.macros`.
@kaxil kaxil mentioned this issue Feb 18, 2025
Merged
@kaxil
Copy link
Member Author

kaxil commented Feb 18, 2025 •
edited
Loading

#46867 should take care of existing/built-in macros and fix your test dag @vatsrahul1001 .

However, user-defined macros, registered via Airflow Plugin (docs link), is still pending as part of this issue

@ashb ashb moved this from Done to Next 2 Weeks in AIP-72 - Task Execution Interface and SDK Feb 24, 2025
@ashb ashb reopened this Feb 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:plugins area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK
Projects
AIP-72 - Task Execution Interface and SDK
Status: Next 2 Weeks
Milestone
Airflow 3.0.0
Development

No branches or pull requests
3 participants
@ashb @kaxil @vatsrahul1001