ARROW-11964: [Rust][DataFusion] Extend constant folding and parquet filtering support #9704

Dandandan · 2021-03-14T16:33:09Z

This PR enables to use some more constant folding and adds support for Between in parquet filtering.

Additionally, it adds some more debug logging.

Before:

[2021-03-14T17:23:20Z DEBUG datafusion::execution::context] Optimized logical plan:
     Projection: #SUM(l_extendedprice Multiply l_discount) AS revenue
      Aggregate: groupBy=[[]], aggr=[[SUM(#l_extendedprice Multiply #l_discount)]]
        Filter: #l_shipdate GtEq CAST(Utf8("1994-01-01") AS Date32) And #l_shipdate Lt CAST(Utf8("1995-01-01") AS Date32) And #l_discount BETWEEN Float64(0.06) Minus Float64(0.01) AND Float64(0.06) Plus Float64(0.01) And #l_quantity Lt Int64(24)
          TableScan: lineitem projection=Some([4, 5, 6, 10]), filters=[#l_shipdate GtEq CAST(Utf8("1994-01-01") AS Date32), #l_shipdate Lt CAST(Utf8("1995-01-01") AS Date32), #l_discount BETWEEN Float64(0.06) Minus Float64(0.01) AND Float64(0.06) Plus Float64(0.01), #l_quantity Lt Int64(24)]

After:

[2021-03-14T17:23:20Z DEBUG datafusion::execution::context] Optimized logical plan:
     Projection: #SUM(l_extendedprice Multiply l_discount) AS revenue
      Aggregate: groupBy=[[]], aggr=[[SUM(#l_extendedprice Multiply #l_discount)]]
        Filter: #l_shipdate GtEq Date32("8766") And #l_shipdate Lt Date32("9131") And #l_discount BETWEEN Float64(0.049999999999999996) AND Float64(0.06999999999999999) And #l_quantity Lt Int64(24)
          TableScan: lineitem projection=Some([4, 5, 6, 10]), filters=[#l_shipdate GtEq Date32("8766"), #l_shipdate Lt Date32("9131"), #l_discount BETWEEN Float64(0.049999999999999996) AND Float64(0.06999999999999999), #l_quantity Lt Int64(24)]

Todo: address/fix some tests

github-actions · 2021-03-14T16:33:27Z

https://issues.apache.org/jira/browse/ARROW-11964

yordan-pavlov · 2021-03-14T21:13:25Z

rust/datafusion/src/physical_plan/parquet.rs

        _ => {
+            debug!("Filter expression not supported in ParquetExec {:?}", expr);


thank you for adding some debug! logs, I have been thinking how predicate push-down into parquet could be displayed into explain output for easier diagnostics, but lately I have been preoccupied with rethinking of reading of arrow arrays from parquet files

Yeah this is very useful! I found that the filter pushdown doesn't filter out anything on the TPC-H benchmark (even when adding some extra support), as the data is distributed so evenly, so every row group contains almost the complete range of values.

I think @andygrove was thinking about some more general support for keeping statistics like this, but some debug logging is something easy to add.

Very interested in your progress with Parquet to Arrow! I was hoping this PR would do something for the tpch benchmark, but seems the filter pushdown is not very useful in this case :)

yes, filter push-down into parquet works best when the filter is over sorted data, if the data is randomly distributed across files it can't do much for performance as you have found

@Dandandan regarding progress on reading arrow arrays from parquet files, I have been able to make some progress and hope to be able to update the jira issue with my latest findings in the next day or two

yordan-pavlov · 2021-03-14T21:36:24Z

@Dandandan I have mixed feelings about this change - I get the performance improvement side, but in my opinion it makes the explain output less readable by replacing the string to date cast with a date: #l_shipdate GtEq CAST(Utf8("1994-01-01") vs #l_shipdate GtEq Date32("8766"); for me the string to date cast is much more readable and I can't really tell what date is Date32("8766")

Dandandan · 2021-03-14T21:46:02Z

@Dandandan I have mixed feelings about this change - I get the performance improvement side, but in my opinion it makes the explain output less readable by replacing the string to date cast with a date: #l_shipdate GtEq CAST(Utf8("1994-01-01") vs #l_shipdate GtEq Date32("8766"); for me the string to date cast is much more readable and I can't really tell what date is Date32("8766")

The Date32("8766") I don't find very elegant, but it's not really part of this PR I would say. The formatting of constant dates could be improved to not show the integer value of the date but a formatted yyyy-mm-dd date instead. I agree the non-folded version is much easier to understand than this integer value. The idea of the PR is of course to evaluate the constant expressions, which in some cases might enable other optimizations.

alamb · 2021-03-16T21:10:52Z

rust/datafusion/src/optimizer/constant_folding.rs

@@ -188,6 +188,97 @@ impl<'a> ExprRewriter for ConstantRewriter<'a> {
                        right,
                    },
                },
+                Operator::Plus => match (left.as_ref(), right.as_ref()) {


returnString · 2021-03-17T21:35:52Z

rust/datafusion/src/optimizer/constant_folding.rs

+                        (Some(l), Some(r)) => {
+                            Expr::Literal(ScalarValue::Int32(Some(l + r)))
+                        }
+                        _ => Expr::Literal(ScalarValue::Int64(None)),


Would it be worth adding some kind of macro to simplify the maintenance of this section? I'm not sure if this is intentionally an Int64 or whether this is meant to be an Int32, and I wonder if we could have a macro that reduces these maths ops to just the ScalarValue variant and an operator (slight awkwardness: can't pass operators to macros).

alamb · 2021-04-19T10:23:27Z

The Apache Arrow Rust community is moving the Rust implementation into its own dedicated github repositories arrow-rs and arrow-datafusion. It is likely we will not merge this PR into this repository

Please see the mailing-list thread for more details

We expect the process to take a few days and will follow up with a migration plan for the in-flight PRs.

jorgecarleitao · 2021-04-30T15:09:28Z

@Dandandan , is this already moved to arrow-rs?

Dandandan · 2021-04-30T15:43:08Z

Not yet, I will add some issues based on the PR and split this thing in two, I think this should be the way to go:

Constant folding should be generalized using the already available expression evaluation code for constants. This to avoid duplication of code and make sure both sides are equivalent/correct well tested.
Parquet predicate pushdown can support more boolean expressions like between / in / etc.

alamb · 2021-05-03T13:01:05Z

#10096 has removed the arrow implementation from this repository (it now resides in https://github.com/apache/arrow-rs and https://github.com/apache/arrow-datafusion) in the hopes of streamlining the development process

Please re-target this PR (let us know if you need help doing so) to one/both of the new repositories.

Thank you for understanding and helping to make arrow-rs and datafusion better

Add more constant folding and parquet filtering support

352da11

github-actions bot added Component: Rust - DataFusion Component: Rust labels Mar 14, 2021

Dandandan added 2 commits March 14, 2021 18:00

Add more debugging

dc1d81c

Remove clone

49009a6

yordan-pavlov reviewed Mar 14, 2021

View reviewed changes

alamb reviewed Mar 16, 2021

View reviewed changes

returnString reviewed Mar 17, 2021

View reviewed changes

alamb closed this May 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-11964: [Rust][DataFusion] Extend constant folding and parquet filtering support #9704

ARROW-11964: [Rust][DataFusion] Extend constant folding and parquet filtering support #9704

Dandandan commented Mar 14, 2021 •

edited

Loading

github-actions bot commented Mar 14, 2021

yordan-pavlov Mar 14, 2021

Dandandan Mar 14, 2021

Dandandan Mar 14, 2021

yordan-pavlov Mar 14, 2021

yordan-pavlov Mar 14, 2021

yordan-pavlov commented Mar 14, 2021

Dandandan commented Mar 14, 2021

alamb Mar 16, 2021

returnString Mar 17, 2021

alamb commented Apr 19, 2021

jorgecarleitao commented Apr 30, 2021

Dandandan commented Apr 30, 2021

alamb commented May 3, 2021

		_ => {
		debug!("Filter expression not supported in ParquetExec {:?}", expr);

ARROW-11964: [Rust][DataFusion] Extend constant folding and parquet filtering support #9704

ARROW-11964: [Rust][DataFusion] Extend constant folding and parquet filtering support #9704

Conversation

Dandandan commented Mar 14, 2021 • edited Loading

github-actions bot commented Mar 14, 2021

yordan-pavlov Mar 14, 2021

Choose a reason for hiding this comment

Dandandan Mar 14, 2021

Choose a reason for hiding this comment

Dandandan Mar 14, 2021

Choose a reason for hiding this comment

yordan-pavlov Mar 14, 2021

Choose a reason for hiding this comment

yordan-pavlov Mar 14, 2021

Choose a reason for hiding this comment

yordan-pavlov commented Mar 14, 2021

Dandandan commented Mar 14, 2021

alamb Mar 16, 2021

Choose a reason for hiding this comment

returnString Mar 17, 2021

Choose a reason for hiding this comment

alamb commented Apr 19, 2021

jorgecarleitao commented Apr 30, 2021

Dandandan commented Apr 30, 2021

alamb commented May 3, 2021

Dandandan commented Mar 14, 2021 •

edited

Loading