[SPARK-6949] [SQL] [PySpark] Support Date/Timestamp in Column expression #5570
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -2267,6 +2267,8 @@ def _prepare_for_python_RDD(sc, command, obj=None): | |||
| # The broadcast will have same life cycle as created PythonRDD | |||
| broadcast = sc.broadcast(pickled_command) | |||
| pickled_command = ser.dumps(broadcast) | |||
| # There is a bug in py4j.java_gateway.JavaClass with auto_convert | |||
There was a problem hiding this comment.
can you document what bug it is?
|
@JoshRosen can you take a look at this? I don't really know the py4j stuff. |
|
Test build #30514 has finished for PR 5570 at commit
|
|
Test build #30515 has finished for PR 5570 at commit
|
|
Test build #30525 has finished for PR 5570 at commit
|
|
Looks like the change broke something in MLlib. |
|
Test build #30541 has finished for PR 5570 at commit
|
|
Test build #30545 has finished for PR 5570 at commit
|
|
Thanks. I'm going to merge this in master. |
nemccarthy
pushed a commit
to nemccarthy/spark
that referenced
this pull request
Jun 19, 2015
This PR enable auto_convert in JavaGateway, then we could register a converter for a given types, for example, date and datetime. There are two bugs related to auto_convert, see [1] and [2], we workaround it in this PR. [1] py4j/py4j#160 [2] py4j/py4j#161 cc rxin JoshRosen Author: Davies Liu <[email protected]> Closes apache#5570 from davies/py4j_date and squashes the following commits: eb4fa53 [Davies Liu] fix tests in python 3 d17d634 [Davies Liu] rollback changes in mllib 2e7566d [Davies Liu] convert tuple into ArrayList ceb3779 [Davies Liu] Update rdd.py 3c373f3 [Davies Liu] support date and datetime by auto_convert cb094ff [Davies Liu] enable auto convert
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR enable auto_convert in JavaGateway, then we could register a converter for a given types, for example, date and datetime.
There are two bugs related to auto_convert, see [1] and [2], we workaround it in this PR.
[1] py4j/py4j#160
[2] py4j/py4j#161
cc @rxin @JoshRosen