[SPARK-23589][SQL] ExternalMapToCatalyst should support interpreted execution #20980

maropu · 2018-04-05T04:05:26Z

What changes were proposed in this pull request?

This pr supported interpreted mode for ExternalMapToCatalyst.

How was this patch tested?

Added tests in ObjectExpressionsSuite.

viirya · 2018-04-05T04:49:01Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala

+  override def eval(input: InternalRow): Any = {
+    val result = child.eval(input)
+    if (result != null) {
+      val mapValue = result.asInstanceOf[Map[Any, Any]]


The external map can be java.util.Map or scala.collection.Map.

SparkQA · 2018-04-05T06:32:12Z

Test build #88924 has finished for PR 20980 at commit 52e76d7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-04-06T11:54:29Z

Test build #88980 has finished for PR 20980 at commit 8783b2b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2018-04-07T01:56:58Z

retest this please

SparkQA · 2018-04-07T05:53:58Z

Test build #89005 has finished for PR 20980 at commit 8783b2b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2018-04-11T02:43:16Z

ping

SparkQA · 2018-04-19T07:05:02Z

Test build #89549 has finished for PR 20980 at commit b13f893.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2018-04-19T09:00:13Z

retest this please

hvanhovell · 2018-04-19T12:53:40Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala

+    val result = child.eval(input)
+    if (result != null) {
+      val (keys, values) = mapCatalystConverter(result)
+      new ArrayBasedMapData(ArrayData.toArrayData(keys), ArrayData.toArrayData(values))


So, this is probably Ok since ArrayData.toArrayData calls new GenericArrayData(any) which calls GenericArrayData.anyToSeq which converts the array into a seq (WrappedArray) which is then converted back into an array. It might copy at the end, but I am not entirely sure.

Can we cut the red tape here and just create GenericArrayData objects on the arrays directly?

ya, I also think you're right. I'll fix.

hvanhovell · 2018-04-19T12:54:15Z

...talyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala

+    }
+  }
+
+  test("SPARK-23589 ExternalMapToCatalyst should support interpreted execution") {


Can you also add a test for a null key?

hvanhovell · 2018-04-19T12:55:25Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala

@@ -357,7 +357,8 @@ object JavaTypeInference {
    }
  }

-  private def serializerFor(inputObject: Expression, typeToken: TypeToken[_]): Expression = {
+  private[catalyst] def serializerFor(
+      inputObject: Expression, typeToken: TypeToken[_]): Expression = {


NIT: put typeToken on a new line?

SparkQA · 2018-04-19T14:03:06Z

Test build #89564 has finished for PR 20980 at commit b13f893.

This patch fails from timeout after a configured wait of `300m`.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-04-20T07:05:01Z

Test build #89606 has finished for PR 20980 at commit ec508cf.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2018-04-20T08:09:36Z

retest this please

SparkQA · 2018-04-20T11:40:53Z

Test build #89619 has finished for PR 20980 at commit ec508cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2018-04-20T12:40:30Z

...talyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala

+    }
+  }
+
+  test("SPARK-23589 ExternalMapToCatalyst should support interpreted execution") {


One more thing, can you please directly add ExternalMapToCatalyst expressions here. Using javaSerializerFor for this is pretty confusing and might cause us to test a different code path at some point.

hvanhovell · 2018-04-20T12:41:10Z

@maropu one more thing, otherwise LGTM.

SparkQA · 2018-04-23T07:05:02Z

Test build #89698 has finished for PR 20980 at commit eaef6b3.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2018-04-23T08:35:12Z

retest this please

SparkQA · 2018-04-23T12:16:11Z

Test build #89709 has finished for PR 20980 at commit eaef6b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2018-04-23T12:20:09Z

...talyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala

+  private def scalaMapSerializerFor[T: TypeTag, U: TypeTag](inputObject: Expression): Expression = {
+    import org.apache.spark.sql.catalyst.ScalaReflection._
+
+    val curId = new java.util.concurrent.atomic.AtomicInteger()


What is this?

hvanhovell · 2018-04-23T12:22:52Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala

+          val entry = iter.next()
+          val (key, value) = (entry.getKey, entry.getValue)
+          keys(i) = if (key != null) {
+            keyConverter.eval(InternalRow.fromSeq(key :: Nil))


Please reuse the InternalRow.

hvanhovell · 2018-04-23T12:23:01Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala

+        var i = 0
+        for ((key, value) <- data) {
+          keys(i) = if (key != null) {
+            keyConverter.eval(InternalRow.fromSeq(key :: Nil))


hvanhovell

LGTM - merging to master. Can you address the nits in a follow-up? Thanks!

maropu · 2018-04-23T13:28:24Z

ok, Thanks!

…yst eval ## What changes were proposed in this pull request? This pr is a follow-up of #20980 and fixes code to reuse `InternalRow` for converting input keys/values in `ExternalMapToCatalyst` eval. ## How was this patch tested? Existing tests. Author: Takeshi Yamamuro <[email protected]> Closes #21137 from maropu/SPARK-23589-FOLLOWUP.

viirya reviewed Apr 5, 2018

View reviewed changes

maropu force-pushed the SPARK-23589 branch from 52e76d7 to 8783b2b Compare April 6, 2018 09:24

maropu force-pushed the SPARK-23589 branch from 8783b2b to b13f893 Compare April 19, 2018 04:10

hvanhovell reviewed Apr 19, 2018

View reviewed changes

maropu force-pushed the SPARK-23589 branch from b13f893 to ec508cf Compare April 20, 2018 04:51

hvanhovell reviewed Apr 20, 2018

View reviewed changes

maropu added 4 commits April 23, 2018 09:59

ExternalMapToCatalyst should support interpreted execution

f3c35bf

Fix

ce00040

Fix

10d89ff

Fix

eaef6b3

maropu force-pushed the SPARK-23589 branch from ec508cf to eaef6b3 Compare April 23, 2018 05:30

hvanhovell reviewed Apr 23, 2018

View reviewed changes

hvanhovell approved these changes Apr 23, 2018

View reviewed changes

asfgit closed this in afbdf42 Apr 23, 2018

maropu mentioned this pull request Apr 24, 2018

[SPARK-23589][SQL][FOLLOW-UP] Reuse InternalRow in ExternalMapToCatalyst eval #21137

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-23589][SQL] ExternalMapToCatalyst should support interpreted execution #20980

[SPARK-23589][SQL] ExternalMapToCatalyst should support interpreted execution #20980

maropu commented Apr 5, 2018

viirya Apr 5, 2018

maropu Apr 5, 2018

SparkQA commented Apr 5, 2018

SparkQA commented Apr 6, 2018

maropu commented Apr 7, 2018

SparkQA commented Apr 7, 2018

maropu commented Apr 11, 2018

SparkQA commented Apr 19, 2018

maropu commented Apr 19, 2018

hvanhovell Apr 19, 2018

maropu Apr 20, 2018

hvanhovell Apr 19, 2018

maropu Apr 20, 2018

hvanhovell Apr 19, 2018

SparkQA commented Apr 19, 2018

SparkQA commented Apr 20, 2018

maropu commented Apr 20, 2018

SparkQA commented Apr 20, 2018

hvanhovell Apr 20, 2018

maropu Apr 20, 2018

hvanhovell commented Apr 20, 2018

SparkQA commented Apr 23, 2018

maropu commented Apr 23, 2018

SparkQA commented Apr 23, 2018

hvanhovell Apr 23, 2018

hvanhovell Apr 23, 2018

hvanhovell Apr 23, 2018

hvanhovell left a comment

maropu commented Apr 23, 2018

[SPARK-23589][SQL] ExternalMapToCatalyst should support interpreted execution #20980

[SPARK-23589][SQL] ExternalMapToCatalyst should support interpreted execution #20980

Conversation

maropu commented Apr 5, 2018

What changes were proposed in this pull request?

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 5, 2018

SparkQA commented Apr 6, 2018

maropu commented Apr 7, 2018

SparkQA commented Apr 7, 2018

maropu commented Apr 11, 2018

SparkQA commented Apr 19, 2018

maropu commented Apr 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 19, 2018

SparkQA commented Apr 20, 2018

maropu commented Apr 20, 2018

SparkQA commented Apr 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvanhovell commented Apr 20, 2018

SparkQA commented Apr 23, 2018

maropu commented Apr 23, 2018

SparkQA commented Apr 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvanhovell left a comment

Choose a reason for hiding this comment

maropu commented Apr 23, 2018