Skip to content

Commit

Permalink
[SPARK-21603][SQL][FOLLOW-UP] Change the default value of maxLinesPer…
Browse files Browse the repository at this point in the history
…Function into 4000

## What changes were proposed in this pull request?
This pr changed the default value of `maxLinesPerFunction` into `4000`. In #18810, we had this new option to disable code generation for too long functions and I found this option only affected `Q17` and `Q66` in TPC-DS. But, `Q66` had some performance regression:

```
Q17 w/o #18810, 3224ms --> q17 w/#18810, 2627ms (improvement)
Q66 w/o #18810, 1712ms --> q66 w/#18810, 3032ms (regression)
```

To keep the previous performance in TPC-DS, we better set higher value at `maxLinesPerFunction` by default.

## How was this patch tested?
Existing tests.

Author: Takeshi Yamamuro <[email protected]>

Closes #19021 from maropu/SPARK-21603-FOLLOWUP-1.
  • Loading branch information
maropu authored and gatorsmile committed Aug 23, 2017
1 parent 1662e93 commit 6942aee
Showing 1 changed file with 3 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -577,10 +577,10 @@ object SQLConf {
.doc("The maximum lines of a single Java function generated by whole-stage codegen. " +
"When the generated function exceeds this threshold, " +
"the whole-stage codegen is deactivated for this subtree of the current query plan. " +
"The default value 2667 is the max length of byte code JIT supported " +
"for a single function(8000) divided by 3.")
"The default value 4000 is the max length of byte code JIT supported " +
"for a single function(8000) divided by 2.")
.intConf
.createWithDefault(2667)
.createWithDefault(4000)

val FILES_MAX_PARTITION_BYTES = buildConf("spark.sql.files.maxPartitionBytes")
.doc("The maximum number of bytes to pack into a single partition when reading files.")
Expand Down

0 comments on commit 6942aee

Please sign in to comment.