[PySpark] Fix tests with Python 2.6 in 1.0 branch #3668

davies · 2014-12-10T20:16:10Z

[SPARK-2951] [PySpark] support unpickle array.array for Python 2.6

Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite.

There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue.

[PySpark] [SPARK-2954] [SPARK-2948] [SPARK-2910] [SPARK-2101] Python 2.6 Fixes

- Modify python/run-tests to test with Python 2.6
- Use unittest2 when running on Python 2.6.
- Fix issue with namedtuple.
- Skip TestOutputFormat.test_newhadoop on Python 2.6 until SPARK-2951 is fixed.
- Fix MLlib _deserialize_double on Python 2.6.

[SPARK-3867][PySpark] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed
./python/run-tests search a Python 2.6 executable on PATH and use it if available.
When using Python 2.6, it is going to import unittest2 module which is not a standard library in Python 2.6, so it fails with Import

Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite. There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue. I had send an PR to Pyrolite to fix it: irmen/Pyrolite#11 Author: Davies Liu <[email protected]> Closes apache#2365 from davies/pickle and squashes the following commits: f44f771 [Davies Liu] enable tests about array 3908f5c [Davies Liu] Merge branch 'master' into pickle c77c87b [Davies Liu] cleanup debugging code 60e4e2f [Davies Liu] support unpickle array.array for Python 2.6 Conflicts: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala python/pyspark/tests.py

SparkQA · 2014-12-10T20:21:09Z

Test build #24318 has started for PR 3668 at commit 7c55cff.

This patch merges cleanly.

SparkQA · 2014-12-10T21:48:58Z

Test build #24318 has finished for PR 3668 at commit 7c55cff.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-10T21:49:03Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24318/
Test FAILed.

JoshRosen · 2014-12-10T21:56:00Z

Looks like we also need to backport #1874 so that the tests can work on Python 2.6?

andrewor14 · 2014-12-10T22:23:58Z

@davies would you mind squeezing all of that into this PR so we can fix the build once and for all?

…2.6 Fixes - Modify python/run-tests to test with Python 2.6 - Use unittest2 when running on Python 2.6. - Fix issue with namedtuple. - Skip TestOutputFormat.test_newhadoop on Python 2.6 until SPARK-2951 is fixed. - Fix MLlib _deserialize_double on Python 2.6. Closes apache#1868. Closes apache#1042. Author: Josh Rosen <[email protected]> Closes apache#1874 from JoshRosen/python2.6 and squashes the following commits: 983d259 [Josh Rosen] [SPARK-2954] Fix MLlib _deserialize_double on Python 2.6. 5d18fd7 [Josh Rosen] [SPARK-2948] [SPARK-2910] [SPARK-2101] Python 2.6 fixes Conflicts: python/pyspark/mllib/_common.py python/pyspark/serializers.py python/pyspark/tests.py

…on 2.6 and unittest2 is not installed ./python/run-tests search a Python 2.6 executable on PATH and use it if available. When using Python 2.6, it is going to import unittest2 module which is not a standard library in Python 2.6, so it fails with ImportError. Author: cocoatomo <[email protected]> Closes apache#2759 from cocoatomo/issues/3867-unittest2-import-error and squashes the following commits: f068eb5 [cocoatomo] [SPARK-3867] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed

SparkQA · 2014-12-10T22:35:36Z

Test build #24324 has started for PR 3668 at commit 14ad3d9.

This patch merges cleanly.

davies · 2014-12-10T22:36:05Z

@andrewor14 I had added those two commits.

SparkQA · 2014-12-10T22:40:28Z

Test build #24326 has started for PR 3668 at commit bda1c72.

This patch merges cleanly.

SparkQA · 2014-12-10T23:55:11Z

Test build #24324 has finished for PR 3668 at commit 14ad3d9.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-10T23:55:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24324/
Test FAILed.

SparkQA · 2014-12-10T23:57:28Z

Test build #24326 has finished for PR 3668 at commit bda1c72.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-10T23:57:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24326/
Test FAILed.

SparkQA · 2014-12-11T00:46:07Z

Test build #24337 has started for PR 3668 at commit b32583d.

This patch merges cleanly.

SparkQA · 2014-12-11T02:26:34Z

Test build #24337 has finished for PR 3668 at commit b32583d.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-11T02:26:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24337/
Test FAILed.

davies · 2014-12-11T05:00:31Z

@andrewor14 The failed test may be related to the version of NumPy, I can not reproduce it with 1.8.1

andrewor14 · 2014-12-11T19:05:24Z

Yeah branch 0.9 is also having the same problem. I haven't looked deep into the issue yet but maybe @shaneknapp has a better idea?

shaneknapp · 2014-12-11T19:49:32Z

would you guys be adverse to me upping the numpy version on the workers? currently, we're running numpy-1.4.1-9.el6.x86_64, which is ancient compared to the L&G release (1.9.1).

anyone know of any backwards-compatibility issues offhand? in the meanwhile, i will set up one of our staging nodes and start testing.

i'd rather not to a system-wide update during a release cycle if at all possible... but if i have to do this, i will bring up some new jenkins workers (which i was going to do during tomorrow morning's maintanence anyways) w/an updated numpy specifically for these builds.

EDIT: just looked at the spark docs and it looks like i'll indeed have to update numpy. :)

andrewor14 · 2014-12-11T20:15:03Z

Yeah I think the case is that in branch 0.9 we still require numpy 1.7 but later branches require only 1.4. For branch-1.0, however, it fails with some other error that I would assume is because we need a more recent version of numpy that is explicitly required.

I think we should just upgrade it, since not super likely that we'll be pushing more commits into the current RC.

shaneknapp · 2014-12-11T22:38:10Z

ok, sgtm. since we're not blocking the release, i'll just get everything staged and do the upgrade during the scheduled maintenance tomorrow morning.

andrewor14 · 2014-12-11T23:29:43Z

Great thanks Shane.

JoshRosen · 2014-12-12T17:49:49Z

Jenkins, retest this please.

SparkQA · 2014-12-12T17:52:43Z

Test build #24412 has started for PR 3668 at commit b32583d.

This patch merges cleanly.

SparkQA · 2014-12-12T19:16:33Z

Test build #24412 has finished for PR 3668 at commit b32583d.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-12T19:16:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24412/
Test FAILed.

JoshRosen · 2014-12-12T19:20:11Z

Jenkins, retest this please.

SparkQA · 2014-12-12T19:22:25Z

Test build #24414 has started for PR 3668 at commit b32583d.

This patch merges cleanly.

SparkQA · 2014-12-12T20:46:43Z

Test build #24414 has finished for PR 3668 at commit b32583d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-12T20:46:48Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24414/
Test PASSed.

shaneknapp · 2014-12-12T21:11:03Z

woot! :)

On Fri, Dec 12, 2014 at 12:47 PM, UCB AMPLab [email protected]
wrote:

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24414/
Test PASSed.

—
Reply to this email directly or view it on GitHub
#3668 (comment).

andrewor14 · 2014-12-13T03:09:10Z

Nice! I'm merging this.

[SPARK-2951] [PySpark] support unpickle array.array for Python 2.6 Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite. There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue. [PySpark] [SPARK-2954] [SPARK-2948] [SPARK-2910] [SPARK-2101] Python 2.6 Fixes - Modify python/run-tests to test with Python 2.6 - Use unittest2 when running on Python 2.6. - Fix issue with namedtuple. - Skip TestOutputFormat.test_newhadoop on Python 2.6 until SPARK-2951 is fixed. - Fix MLlib _deserialize_double on Python 2.6. [SPARK-3867][PySpark] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed ./python/run-tests search a Python 2.6 executable on PATH and use it if available. When using Python 2.6, it is going to import unittest2 module which is not a standard library in Python 2.6, so it fails with Import Author: cocoatomo <[email protected]> Author: Josh Rosen <[email protected]> Author: Davies Liu <[email protected]> Author: Davies Liu <[email protected]> Closes #3668 from davies/port_2365 and squashes the following commits: b32583d [Davies Liu] rollback _common.py bda1c72 [cocoatomo] [SPARK-3867][PySpark] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed 14ad3d9 [Josh Rosen] [PySpark] [SPARK-2954] [SPARK-2948] [SPARK-2910] [SPARK-2101] Python 2.6 Fixes 7c55cff [Davies Liu] [SPARK-2951] [PySpark] support unpickle array.array for Python 2.6

JoshRosen · 2014-12-16T00:49:11Z

Do you mind closing this PR since Andrew has merged it into branch-1.0 (9c21316)? GitHub can't auto-close it since it's not against the master branch.

[SPARK-2951] [PySpark] support unpickle array.array for Python 2.6 Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite. There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue. [PySpark] [SPARK-2954] [SPARK-2948] [SPARK-2910] [SPARK-2101] Python 2.6 Fixes - Modify python/run-tests to test with Python 2.6 - Use unittest2 when running on Python 2.6. - Fix issue with namedtuple. - Skip TestOutputFormat.test_newhadoop on Python 2.6 until SPARK-2951 is fixed. - Fix MLlib _deserialize_double on Python 2.6. [SPARK-3867][PySpark] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed ./python/run-tests search a Python 2.6 executable on PATH and use it if available. When using Python 2.6, it is going to import unittest2 module which is not a standard library in Python 2.6, so it fails with Import Author: cocoatomo <[email protected]> Author: Josh Rosen <[email protected]> Author: Davies Liu <[email protected]> Author: Davies Liu <[email protected]> Closes apache#3668 from davies/port_2365 and squashes the following commits: b32583d [Davies Liu] rollback _common.py bda1c72 [cocoatomo] [SPARK-3867][PySpark] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed 14ad3d9 [Josh Rosen] [PySpark] [SPARK-2954] [SPARK-2948] [SPARK-2910] [SPARK-2101] Python 2.6 Fixes 7c55cff [Davies Liu] [SPARK-2951] [PySpark] support unpickle array.array for Python 2.6 Conflicts: python/pyspark/mllib/tests.py python/pyspark/tests.py python/run-tests

[PySpark] [SPARK-2954] [SPARK-2948] [SPARK-2910] [SPARK-2101] Python 2.6 Fixes - Modify python/run-tests to test with Python 2.6 - Use unittest2 when running on Python 2.6. Author: Josh Rosen <joshrosenapache.org> Closes #3668 from davies/port_2365 and squashes the following commits: Author: cocoatomo <[email protected]> Closes #3968 from davies/fix_python_tests and squashes the following commits: ac4a353 [cocoatomo] [PySpark] Fix tests with Python 2.6 in 1.0 branch

JoshRosen and others added 2 commits December 10, 2014 14:30

davies changed the title ~~[SPARK-2951] [PySpark] support unpickle array.array for Python 2.6 (port to 1.0)~~ [PySpark] Fix tests with Python 2.6 in 1.0 branch Dec 10, 2014

andrewor14 mentioned this pull request Dec 10, 2014

[SPARK-4006] In long running contexts, we encountered the situation of d... #2914

Closed

rollback _common.py

b32583d

shaneknapp mentioned this pull request Dec 11, 2014

[SPARK-4006] Block Manager - Double Register Crash #2854

Closed

davies closed this Dec 16, 2014

davies mentioned this pull request Jan 9, 2015

[PySpark] Fix tests with Python 2.6 in 0.9 branch #3968

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PySpark] Fix tests with Python 2.6 in 1.0 branch #3668

[PySpark] Fix tests with Python 2.6 in 1.0 branch #3668

davies commented Dec 10, 2014

SparkQA commented Dec 10, 2014

SparkQA commented Dec 10, 2014

AmplabJenkins commented Dec 10, 2014

JoshRosen commented Dec 10, 2014

andrewor14 commented Dec 10, 2014

SparkQA commented Dec 10, 2014

davies commented Dec 10, 2014

SparkQA commented Dec 10, 2014

SparkQA commented Dec 10, 2014

AmplabJenkins commented Dec 10, 2014

SparkQA commented Dec 10, 2014

AmplabJenkins commented Dec 10, 2014

SparkQA commented Dec 11, 2014

SparkQA commented Dec 11, 2014

AmplabJenkins commented Dec 11, 2014

davies commented Dec 11, 2014

andrewor14 commented Dec 11, 2014

shaneknapp commented Dec 11, 2014

andrewor14 commented Dec 11, 2014

shaneknapp commented Dec 11, 2014

andrewor14 commented Dec 11, 2014

JoshRosen commented Dec 12, 2014

SparkQA commented Dec 12, 2014

SparkQA commented Dec 12, 2014

AmplabJenkins commented Dec 12, 2014

JoshRosen commented Dec 12, 2014

SparkQA commented Dec 12, 2014

SparkQA commented Dec 12, 2014

AmplabJenkins commented Dec 12, 2014

shaneknapp commented Dec 12, 2014

andrewor14 commented Dec 13, 2014

JoshRosen commented Dec 16, 2014

[PySpark] Fix tests with Python 2.6 in 1.0 branch #3668

[PySpark] Fix tests with Python 2.6 in 1.0 branch #3668

Conversation

davies commented Dec 10, 2014

SparkQA commented Dec 10, 2014

SparkQA commented Dec 10, 2014

AmplabJenkins commented Dec 10, 2014

JoshRosen commented Dec 10, 2014

andrewor14 commented Dec 10, 2014

SparkQA commented Dec 10, 2014

davies commented Dec 10, 2014

SparkQA commented Dec 10, 2014

SparkQA commented Dec 10, 2014

AmplabJenkins commented Dec 10, 2014

SparkQA commented Dec 10, 2014

AmplabJenkins commented Dec 10, 2014

SparkQA commented Dec 11, 2014

SparkQA commented Dec 11, 2014

AmplabJenkins commented Dec 11, 2014

davies commented Dec 11, 2014

andrewor14 commented Dec 11, 2014

shaneknapp commented Dec 11, 2014

andrewor14 commented Dec 11, 2014

shaneknapp commented Dec 11, 2014

andrewor14 commented Dec 11, 2014

JoshRosen commented Dec 12, 2014

SparkQA commented Dec 12, 2014

SparkQA commented Dec 12, 2014

AmplabJenkins commented Dec 12, 2014

JoshRosen commented Dec 12, 2014

SparkQA commented Dec 12, 2014

SparkQA commented Dec 12, 2014

AmplabJenkins commented Dec 12, 2014

shaneknapp commented Dec 12, 2014

andrewor14 commented Dec 13, 2014

JoshRosen commented Dec 16, 2014