Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-23914][SQL][follow-up] refactor ArrayUnion #21937

Closed
wants to merge 4 commits into from
Closed

[SPARK-23914][SQL][follow-up] refactor ArrayUnion #21937

wants to merge 4 commits into from

Conversation

kiszk
Copy link
Member

@kiszk kiszk commented Jul 31, 2018

What changes were proposed in this pull request?

This PR refactors ArrayUnion based on this suggestion.

  1. Generate optimized code for all of the primitive types except boolean
  2. Generate code using ArrayBuilder or ArrayBuffer
  3. Leave only a generic path in the interpreted path

How was this patch tested?

Existing tests

@holdensmagicalunicorn
Copy link

@kiszk, thanks! I am a bot who has found some folks who might be able to help with the review:@ueshin, @rxin and @colorant

@SparkQA
Copy link

SparkQA commented Jul 31, 2018

Test build #93840 has finished for PR 21937 at commit 738a4a0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class OpenHashSet[@specialized(Long, Int, Double, Float) T: ClassTag](
  • sealed class Hasher[@specialized(Long, Int, Double, Float) T] extends Serializable
  • class DoubleHasher extends Hasher[Double]
  • class FloatHasher extends Hasher[Float]
  • case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike

@kiszk kiszk force-pushed the SPARK-23914-follow branch from 738a4a0 to f48dde0 Compare August 4, 2018 07:28
@SparkQA
Copy link

SparkQA commented Aug 4, 2018

Test build #94208 has finished for PR 21937 at commit f48dde0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

ueshin
ueshin reviewed Aug 6, 2018
View reviewed changes
...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala Outdated
|$declareNullTrackVariables
|int $size = 0;
|$arrayBuilderClass $builder =
| ($arrayBuilderClass)$arrayBuilder.make($arrayBuilderClassTag);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: new $arrayBuilderClass() should work?

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala Outdated
val openHashSet = classOf[OpenHashSet[_]].getName
val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$hsTypeName()"
val hashSet = ctx.freshName("hashSet")
val arrayBuilder = "scala.collection.mutable.ArrayBuilder"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: classOf[mutable.ArrayBuilder[_]].getName?

@ueshin
Copy link
Member

ueshin commented Aug 6, 2018

I'll revisit after the conflict is fixed.

@kiszk
Copy link
Member Author

kiszk commented Aug 6, 2018

I see. Now, I am rebasing and resolving conflicts.

@kiszk kiszk force-pushed the SPARK-23914-follow branch from f48dde0 to 93ec9ec Compare August 6, 2018 15:35
@kiszk kiszk changed the title [WIP][SPARK-23914][SQL][follow-up] refactor ArrayUnion [SPARK-23914][SQL][follow-up] refactor ArrayUnion Aug 6, 2018
ueshin
ueshin reviewed Aug 6, 2018
View reviewed changes
...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$hsTypeName()"
val hashSet = ctx.freshName("hashSet")
val arrayBuilder = classOf[mutable.ArrayBuilder[_]].getName
val arrayBuilderClass = s"$arrayBuilder$$of$ptName"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not needed any more?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we still need this to create an intermediate result array. The array is allocated at L3907 as other functions do.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry, I misread.

@kiszk kiszk mentioned this pull request Aug 6, 2018
Closed
@SparkQA
Copy link

SparkQA commented Aug 6, 2018

Test build #94290 has finished for PR 21937 at commit 93ec9ec.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 6, 2018

Test build #94304 has finished for PR 21937 at commit 26c1f8a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@ueshin
Copy link
Member

ueshin commented Aug 7, 2018

LGTM.

@ueshin
Copy link
Member

ueshin commented Aug 7, 2018

Thanks! merging to master.

@asfgit asfgit closed this in 4446a0b Aug 7, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ueshin ueshin ueshin left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kiszk @holdensmagicalunicorn @SparkQA @ueshin