[SPARK-5683] [SQL] Avoid multiple json generator created #4468
Conversation
|
Test build #27092 has started for PR 4468 at commit
|
chenghao-intel
force-pushed
the
json
branch
2 times, most recently
from
ebeb0b3 to
aeb7801
Compare
|
Test build #27093 has started for PR 4468 at commit
|
|
Test build #27092 has finished for PR 4468 at commit
|
|
Test PASSed. |
|
Test build #27093 has finished for PR 4468 at commit
|
|
Test PASSed. |
| iter.map(JsonRDD.rowToJSON(rowSchema, jsonFactory)) | ||
| val writer = new CharArrayWriter() | ||
| // create the Generator without separator inserted between 2 records | ||
| val gen = new JsonFactory().createGenerator(writer).setRootValueSeparator(null) |
There was a problem hiding this comment.
Should we expect any test result change after using setRootValueSeparator(null)?
There was a problem hiding this comment.
A (single space) will be inserted between 2 json records (in string) by default;
Previously, we create thegeneratorobject for each of record in serialization, hence it's OK with default object separator.
However, in this PR, we are trying to using a singlegenerator for all of the records, that's why we need to set it as null, which means nothing will be inserted between records.
|
@chenghao-intel Thank you for the PR. It will be great if you can also attach some performance results. |
|
@yhuai, thanks for reviewing, in a local testing, it shows about 10-15% performance gain. |
|
LGTM |
|
@marmbrus any more comment on this? |
Author: Cheng Hao <[email protected]> Closes #4468 from chenghao-intel/json and squashes the following commits: aeb7801 [Cheng Hao] avoid multiple json generator created (cherry picked from commit a60aea8) Signed-off-by: Michael Armbrust <[email protected]>
|
Thanks, merged to master and 1.3 |
No description provided.