[SPARK-23913][SQL] Add array_intersect function #21102

kiszk · 2018-04-18T16:09:20Z

What changes were proposed in this pull request?

The PR adds the SQL function array_intersect. The behavior of the function is based on Presto's one.

This function returns returns an array of the elements in the intersection of array1 and array2.

Note: The order of elements in the result is not defined.

How was this patch tested?

Added UTs

SparkQA · 2018-04-18T19:39:18Z

Test build #89517 has finished for PR 21102 at commit 548a4b8.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class ArraySetUtils extends BinaryExpression with ExpectsInputTypes
case class ArrayIntersect(left: Expression, right: Expression) extends ArraySetUtils

SparkQA · 2018-04-18T21:31:06Z

Test build #89527 has finished for PR 21102 at commit 2602f8e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-04-19T03:59:49Z

Test build #89538 has finished for PR 21102 at commit 038e98c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2018-04-19T22:06:51Z

cc @ueshin

SparkQA · 2018-04-21T07:02:26Z

Test build #89671 has finished for PR 21102 at commit cd56b7d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class ArraySetUtils extends BinaryExpression with ExpectsInputTypes

SparkQA · 2018-06-26T05:53:47Z

Test build #92323 has finished for PR 21102 at commit cd56b7d.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
abstract class ArraySetUtils extends BinaryExpression with ExpectsInputTypes

SparkQA · 2018-07-10T11:30:44Z

Test build #92813 has finished for PR 21102 at commit 59ea8e2.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class ArraySetLike extends BinaryArrayExpressionWithImplicitCast
case class ArrayIntersect(left: Expression, right: Expression) extends ArraySetLike

SparkQA · 2018-07-10T11:50:41Z

Test build #92814 has finished for PR 21102 at commit 346274d.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-10T16:30:36Z

Test build #92818 has finished for PR 21102 at commit cf7a27c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-11T03:06:20Z

Test build #92833 has finished for PR 21102 at commit 905c31a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-12T19:25:02Z

Test build #92943 has finished for PR 21102 at commit fdc2f6c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-12T23:34:02Z

Test build #92947 has finished for PR 21102 at commit 7d789e2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-13T13:41:54Z

Test build #92970 has finished for PR 21102 at commit fce9eb0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-14T07:05:01Z

Test build #92998 has finished for PR 21102 at commit 5492572.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2018-07-14T13:50:45Z

retest this please

SparkQA · 2018-07-14T18:21:34Z

Test build #93002 has finished for PR 21102 at commit 5492572.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2018-07-16T04:12:32Z

cc @ueshin

ueshin · 2018-07-16T09:28:56Z

core/src/test/scala/org/apache/spark/util/collection/OpenHashSetSuite.scala

+    assert(!set.contains(1132))
+    assert(!set.contains(10000))
+
+    set.remove(999)


What if we remove 999 before 1132? Can we still find 1132?

good catch, I addressed this.

SparkQA · 2018-07-16T11:52:23Z

Test build #93097 has finished for PR 21102 at commit 05a612b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike

SparkQA · 2018-07-16T22:22:02Z

Test build #93129 has finished for PR 21102 at commit 28e0c45.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2018-07-17T01:06:58Z

retest this please

SparkQA · 2018-07-17T04:50:41Z

Test build #93146 has finished for PR 21102 at commit 28e0c45.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2018-07-17T06:20:15Z

retest this please

SparkQA · 2018-07-17T07:05:01Z

Test build #93155 has finished for PR 21102 at commit 28e0c45.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2018-07-17T07:25:02Z

retest this please

SparkQA · 2018-07-17T11:52:37Z

Test build #93157 has finished for PR 21102 at commit 28e0c45.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-04T17:03:17Z

Test build #94213 has finished for PR 21102 at commit 6fba1ee.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ArrayIntersect(left: Expression, right: Expression) extends ArraySetLike

kiszk · 2018-08-04T20:21:41Z

retest this please

SparkQA · 2018-08-04T23:51:53Z

Test build #94221 has finished for PR 21102 at commit 6fba1ee.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ArrayIntersect(left: Expression, right: Expression) extends ArraySetLike

kiszk · 2018-08-05T00:22:18Z

retest this please

SparkQA · 2018-08-05T04:12:33Z

Test build #94231 has finished for PR 21102 at commit 6fba1ee.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ArrayIntersect(left: Expression, right: Expression) extends ArraySetLike

kiszk · 2018-08-05T05:15:23Z

cc @ueshin @cloud-fan

maropu · 2018-08-06T01:39:23Z

sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala

+    checkAnswer(df5.selectExpr("array_intersect(a, b)"), ans5)
+
+    val df6 = Seq((null, null)).toDF("a", "b")
+    intercept[AnalysisException] {


Could you also check the error message?

ueshin

LGTM except for comments and #21102 (comment) and #21102 (comment).

ueshin · 2018-08-06T03:08:48Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+  @transient lazy val evalIntersect: (ArrayData, ArrayData) => ArrayData = {
+    if (elementTypeSupportEquals) {
+      (array1, array2) =>
+        val hs = new OpenHashSet[Any]


How about shortcutting to return an empty array when we find one of the two is empty?

ueshin · 2018-08-06T03:21:24Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+           |  $writeArray2ToHashSet
+           |}
+           |$arrayBuilderClass $builder =
+           |  ($arrayBuilderClass)$arrayBuilder.make($arrayBuilderClassTag);


nit: new $arrayBuilderClass() should work?

ueshin · 2018-08-06T03:30:18Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+        val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$hsTypeName()"
+        val hashSet = ctx.freshName("hashSet")
+        val hashSetResult = ctx.freshName("hashSetResult")
+        val arrayBuilder = "scala.collection.mutable.ArrayBuilder"


nit: classOf[mutable.ArrayBuilder[_]].getName?

ueshin · 2018-08-06T06:09:40Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+          }
+          new GenericArrayData(arrayBuffer)
+        } else {
+          new GenericArrayData(Seq.empty)


nit: Array.empty or Array.emptyObjectArray?

SparkQA · 2018-08-06T07:05:01Z

Test build #94266 has finished for PR 21102 at commit 33781b6.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-06T07:05:02Z

Test build #94264 has finished for PR 21102 at commit ce755e2.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2018-08-06T07:22:57Z

Jenkins, retest this please.

SparkQA · 2018-08-06T11:27:23Z

Test build #94267 has finished for PR 21102 at commit 33781b6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2018-08-06T14:26:50Z

Thanks! merging to master.

cloud-fan · 2018-08-06T16:03:10Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+  }
+
+  override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+    val arrayData = classOf[ArrayData].getName


nit: ArrayData is imported by default in codegen.

Thanks. I will address at #21937

kiszk force-pushed the SPARK-23913 branch from 038e98c to cd56b7d Compare April 21, 2018 03:21

ueshin mentioned this pull request Apr 26, 2018

[SPARK-23922][SQL] Add arrays_overlap function #21028

Closed

kiszk force-pushed the SPARK-23913 branch from cd56b7d to 59ea8e2 Compare July 10, 2018 11:24

kiszk force-pushed the SPARK-23913 branch from 905c31a to fdc2f6c Compare July 12, 2018 19:18

ueshin reviewed Jul 16, 2018

View reviewed changes

kiszk added 8 commits August 4, 2018 22:21

address review comment

352743f

avoid to use OpenHashSet.remove method

89a828b

update dataType setting

90fecd5

update

663358e

optimize nullchecks in generated code

3ef99b8

update

3462b98

refactor methods that are used in Except and Intersect

cf742d4

update

6fba1ee

kiszk force-pushed the SPARK-23913 branch from ab9aa10 to 6fba1ee Compare August 4, 2018 13:39

maropu reviewed Aug 6, 2018

View reviewed changes

ueshin reviewed Aug 6, 2018

View reviewed changes

address review comments

ce755e2

ueshin reviewed Aug 6, 2018

View reviewed changes

address review comment

33781b6

asfgit closed this in 1a5e460 Aug 6, 2018

cloud-fan reviewed Aug 6, 2018

View reviewed changes

zeen mentioned this pull request Apr 12, 2021

Array functions and order preservation prestodb/presto#15924

Open

[SPARK-23913][SQL] Add array_intersect function #21102

[SPARK-23913][SQL] Add array_intersect function #21102

Conversation

kiszk commented Apr 18, 2018

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Apr 18, 2018

SparkQA commented Apr 18, 2018

SparkQA commented Apr 19, 2018

gatorsmile commented Apr 19, 2018

SparkQA commented Apr 21, 2018

SparkQA commented Jun 26, 2018

SparkQA commented Jul 10, 2018

SparkQA commented Jul 10, 2018

SparkQA commented Jul 10, 2018

SparkQA commented Jul 11, 2018

SparkQA commented Jul 12, 2018

SparkQA commented Jul 12, 2018

SparkQA commented Jul 13, 2018

SparkQA commented Jul 14, 2018

kiszk commented Jul 14, 2018

SparkQA commented Jul 14, 2018

kiszk commented Jul 16, 2018

ueshin Jul 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 16, 2018

SparkQA commented Jul 16, 2018

kiszk commented Jul 17, 2018

SparkQA commented Jul 17, 2018

kiszk commented Jul 17, 2018

SparkQA commented Jul 17, 2018

kiszk commented Jul 17, 2018

SparkQA commented Jul 17, 2018

SparkQA commented Aug 4, 2018

kiszk commented Aug 4, 2018

SparkQA commented Aug 4, 2018

kiszk commented Aug 5, 2018

SparkQA commented Aug 5, 2018

kiszk commented Aug 5, 2018

Choose a reason for hiding this comment

ueshin left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Aug 6, 2018

SparkQA commented Aug 6, 2018

ueshin commented Aug 6, 2018

SparkQA commented Aug 6, 2018

ueshin commented Aug 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ueshin Jul 16, 2018 •

edited

Loading

ueshin left a comment •

edited

Loading