Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot pushdown filters on types not native to Substrait #2972

Open
westonpace opened this issue Oct 3, 2024 · 0 comments
Open

Cannot pushdown filters on types not native to Substrait #2972

westonpace opened this issue Oct 3, 2024 · 0 comments

Comments

@westonpace
Copy link
Contributor

The arrow type system is wider than Substrait's default types (e.g. unsigned integers, large strings, etc.). There is a spec for expressing these types as Substrait user defined types (https://github.com/apache/arrow/blob/main/format/substrait/extension_types.yaml) however, Datafusion doesn't yet support the extension.

As a result, filters containing these types will be rejected when used as pushdown (e.g. through DuckDb) or provided directly from Substrait.

This is primarily a tracking issue. Once Datafusion supports these types we shouldn't need to do any additional work but it would be good to add some regression tests.

westonpace added a commit that referenced this issue Oct 25, 2024
This is particularly useful for operations like "delete by id"

I ran into a fair bit of difficulty with this PR and punted a few things
to follow-ups (#2971 #2972).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant