Skip to content

Commit

Permalink
[SPARK-22465][CORE][FOLLOWUP] Use existing partitioner when defaultNu…
Browse files Browse the repository at this point in the history
…mPartitions is equal to maxPartitioner.numPartitions

## What changes were proposed in this pull request?

Followup of apache#20091. We could also use existing partitioner when defaultNumPartitions is equal to the maxPartitioner's numPartitions.

## How was this patch tested?

Existed.

Closes apache#23581 from Ngone51/dev-use-existing-partitioner-when-defaultNumPartitions-equalTo-MaxPartitioner#-numPartitions.

Authored-by: Ngone51 <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
  • Loading branch information
Ngone51 authored and jackylee-ch committed Feb 18, 2019
1 parent ef9a4d3 commit f9984d1
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions core/src/main/scala/org/apache/spark/Partitioner.scala
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ object Partitioner {
*
* When available, we choose the partitioner from rdds with maximum number of partitions. If this
* partitioner is eligible (number of partitions within an order of maximum number of partitions
* in rdds), or has partition number higher than default partitions number - we use this
* partitioner.
* in rdds), or has partition number higher than or equal to default partitions number - we use
* this partitioner.
*
* Otherwise, we'll use a new HashPartitioner with the default partitions number.
*
Expand All @@ -79,9 +79,9 @@ object Partitioner {
}

// If the existing max partitioner is an eligible one, or its partitions number is larger
// than the default number of partitions, use the existing partitioner.
// than or equal to the default number of partitions, use the existing partitioner.
if (hasMaxPartitioner.nonEmpty && (isEligiblePartitioner(hasMaxPartitioner.get, rdds) ||
defaultNumPartitions < hasMaxPartitioner.get.getNumPartitions)) {
defaultNumPartitions <= hasMaxPartitioner.get.getNumPartitions)) {
hasMaxPartitioner.get.partitioner.get
} else {
new HashPartitioner(defaultNumPartitions)
Expand Down

0 comments on commit f9984d1

Please sign in to comment.