[SPARK-23312][SQL][followup] add a config to turn off vectorized cache reader #20513
Conversation
| @@ -61,6 +61,9 @@ case class InMemoryTableScanExec( | |||
| }) && !WholeStageCodegenExec.isTooManyFields(conf, relation.schema) | |||
| } | |||
|
|
|||
| // TODO: revisit this. Shall we always turn off whole stage codegen if the output data are rows? | |||
| override def supportCodegen: Boolean = supportsBatch | |||
There was a problem hiding this comment.
In 2.4 we should look into this. My gut feeling is we don't need to enable whole stage codegen for scan nodes that output data as rows.
There was a problem hiding this comment.
Yeah, we can do more perf measurement after 2.3 release
There was a problem hiding this comment.
I think this is safe to keep the same behavior of 2.2.
I'm not sure if enabling whole stage codegen can hurt performance for scan nodes, btw. We can revisit this, of course.
There was a problem hiding this comment.
We confirmed that we got performance improvements in several cases.
I will revisit this with more cases after 2.3 release.
|
LGTM pending Jenkins. |
|
Test build #87088 has finished for PR 20513 at commit
|
|
retest this please |
|
Test build #87093 has finished for PR 20513 at commit
|
|
retest this please. |
|
LGTM |
|
Test build #87097 has finished for PR 20513 at commit
|
|
Test build #87108 has finished for PR 20513 at commit
|
|
retest this please |
|
Test build #87113 has finished for PR 20513 at commit
|
Seems like some environmental problems, let's try again. |
|
retest this please |
|
Test build #87118 has finished for PR 20513 at commit
|
|
Thanks! Merged to master/2.3 |
…e reader ## What changes were proposed in this pull request? #20483 tried to provide a way to turn off the new columnar cache reader, to restore the behavior in 2.2. However even we turn off that config, the behavior is still different than 2.2. If the output data are rows, we still enable whole stage codegen for the scan node, which is different with 2.2, we should also fix it. ## How was this patch tested? existing tests. Author: Wenchen Fan <[email protected]> Closes #20513 from cloud-fan/cache. (cherry picked from commit ac7454c) Signed-off-by: gatorsmile <[email protected]>
…e reader ## What changes were proposed in this pull request? apache#20483 tried to provide a way to turn off the new columnar cache reader, to restore the behavior in 2.2. However even we turn off that config, the behavior is still different than 2.2. If the output data are rows, we still enable whole stage codegen for the scan node, which is different with 2.2, we should also fix it. ## How was this patch tested? existing tests. Author: Wenchen Fan <[email protected]> Closes apache#20513 from cloud-fan/cache.
What changes were proposed in this pull request?
#20483 tried to provide a way to turn off the new columnar cache reader, to restore the behavior in 2.2. However even we turn off that config, the behavior is still different than 2.2.
If the output data are rows, we still enable whole stage codegen for the scan node, which is different with 2.2, we should also fix it.
How was this patch tested?
existing tests.