Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-28699][SQL] Disable using radix sort for ShuffleExchangeExec in repartition case #25491

Closed
wants to merge 6 commits into from
Closed

[SPARK-28699][SQL] Disable using radix sort for ShuffleExchangeExec in repartition case #25491

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1256c87
[SPARK-28699][Core] Disable using radix sort for ShuffleExchangeExec …
xuanyuanking Aug 19, 2019
398a891
More comments
xuanyuanking Aug 19, 2019
ec57677
comment address
xuanyuanking Aug 20, 2019
8bf4cc8
fix comment
xuanyuanking Aug 20, 2019
c692265
Optimization by using radix sort if possible
xuanyuanking Aug 20, 2019
2e26335
Revert "Optimization by using radix sort if possible"
xuanyuanking Aug 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,7 @@ object ShuffleExchangeExec {
}
// The comparator for comparing row hashcode, which should always be Integer.
val prefixComparator = PrefixComparators.LONG
val canUseRadixSort = SQLConf.get.enableRadixSort

// The prefix computer generates row hashcode as the prefix, so we may decrease the
// probability that the prefixes are equal when input rows choose column values from a
// limited range.
Expand All @@ -264,7 +264,9 @@ object ShuffleExchangeExec {
prefixComparator,
prefixComputer,
pageSize,
canUseRadixSort)
// We are comparing binary here, which does not support radix sort.
// See more details in SPARK-28699.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about

we are comparing binary here, which does not support radix sort. See more details in SPARK-28699.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy, shorten it in 8bf4cc8.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @cloud-fan . Is this the only one for this topic? Or, do we have more TODO items against old branches (branch-2.4 and branch-2.3)?

Copy link
Contributor

@cloud-fan cloud-fan Aug 20, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have one more: #25498 , which will be merged soon, after tests pass.

The current PR may need more time. We need to collect feedback about whether we should disable "sort before repartition" by default. cc @jiangxb1987

false)
sorter.sort(iter.asInstanceOf[Iterator[UnsafeRow]])
}
} else {
Expand Down