[SPARK-5738] [SQL] Reuse mutable row for each record at jsonStringToRow

[yanboliang]

When serialize json string to row, reuse a mutable row for both each record and inner nested structure instead of creating a new one each time.

[SparkQA]

Test build #27284 has started for PR 4527 at commit b0c2b14.

• This patch merges cleanly.

[yanboliang]

https://issues.apache.org/jira/browse/SPARK-5738

[SparkQA]

Test build #27284 has finished for PR 4527 at commit b0c2b14.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27284/
Test FAILed.

[yanboliang]

retest this please

[SparkQA]

Test build #27288 has started for PR 4527 at commit b0c2b14.

• This patch merges cleanly.

[SparkQA]

Test build #27288 has finished for PR 4527 at commit b0c2b14.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27288/
Test PASSed.

[chenghao-intel]

Move this inside of mapPartitions, to reduce the closure serialization overhead. And I didn't see any benefit when using the SpecificMutableRow, why not just use the GenericMutableRow instead?

[yanboliang]

You are right, it's not appropriate to use SpecificMutableRow here. I will change back to GenericMutableRow.

[yhuai]

Thank you for working on it.

Seems new SpecificMutableRow(schema.fields.map(_.dataType)) cannot handle nested structure. I think we need to use the schema to create the top level mutable row and all inner rows (for inner StructType).

[yhuai]

Also, can you add performance numbers?

[yhuai]

Oh, enforceCorrectType will take care inner structures by calling asRow.

It will be great if we can use mutable rows for inner structures as well.

[yanboliang]

@chenghao-intel @yhuai
Thank you for your advice and it's very useful.
We can use mutable rows for both top level records and inner structures at present.

[SparkQA]

Test build #27351 has started for PR 4527 at commit c30a358.

• This patch merges cleanly.

[SparkQA]

Test build #27351 has finished for PR 4527 at commit c30a358.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27351/
Test PASSed.

[SparkQA]

Test build #27513 has started for PR 4527 at commit 6cd26fe.

• This patch merges cleanly.

[SparkQA]

Test build #27513 has finished for PR 4527 at commit 6cd26fe.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27513/
Test FAILed.

[SparkQA]

Test build #27514 has started for PR 4527 at commit 7039fa7.

• This patch merges cleanly.

[SparkQA]

Test build #27514 has finished for PR 4527 at commit 7039fa7.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27514/
Test FAILed.

[SparkQA]

Test build #27522 has started for PR 4527 at commit 2d45c68.

• This patch merges cleanly.

[SparkQA]

Test build #27524 has started for PR 4527 at commit 2286ac5.

• This patch merges cleanly.

[SparkQA]

Test build #27524 has finished for PR 4527 at commit 2286ac5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27524/
Test PASSed.

[SparkQA]

Test build #27522 has finished for PR 4527 at commit 2d45c68.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27522/
Test PASSed.

[yanboliang]

This improvement is very similar to #758, so I have run the similar performance test.
The benchmark suggests this optimization made the optimized version about 1.5x faster when scanning JSON table, but it depends on the JSON schema especially for whether different records have different schema.
For a JSON file with 188010 lines, the build scan consumed time is:
original: Takes 15598 ms
optimized: Takes 10152 ms

[yanboliang]

@liancheng @rxin @marmbrus , can you review it ?

[rxin]

Thanks - sorry for not having looked at this earlier. Do you see any performance gains with this change? My understanding is that JSON is already very slow, and thus the code path is hard to optimize.