Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(rust,python): optimize away nested unions in lp #7861

Merged
merged 5 commits into from
Apr 3, 2023

Conversation

universalmind303
Copy link
Collaborator

Implementation was mostly taken from the SimplifyExprRule for flattening nested concat_str calls.

This'll flatten any arbitrarily nested unions into a single union.

closes #7855

@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars labels Mar 29, 2023
Copy link
Member

@ritchie46 ritchie46 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I have left one remark.


match lp {
Union { inputs, options }
if inputs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a state flattened_by_opt to the UnionOptions. This will prevent we run this loop every time this rule is checked. It is not uncommon to have unions over a lot of files, so it is worth it to return early in such a case.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure I 100% follow. If we add a flattened_by_opt when would we set it? and when would we return early if it were set?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would set it in this optimization and in this optimization we return early.

These optimization rules can be called many times (until fixed point).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ritchie46, I added the opt state flattened_by_opt. This should be ready for final review.

@universalmind303
Copy link
Collaborator Author

@ritchie46 The test failures seem unrelated to my work. I already rebased from master, So i'm not entirely sure why those tests are failing.

@ritchie46 ritchie46 merged commit d24cf6a into pola-rs:master Apr 3, 2023
@ritchie46
Copy link
Member

Yeap, test failure is unrelated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimize nested unions within LogicalPlan
2 participants