From ac4a353e1d514379f24298644c7c141536ec415e Mon Sep 17 00:00:00 2001 From: cocoatomo Date: Fri, 12 Dec 2014 19:10:12 -0800 Subject: [PATCH] [PySpark] Fix tests with Python 2.6 in 1.0 branch [SPARK-2951] [PySpark] support unpickle array.array for Python 2.6 Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite. There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue. [PySpark] [SPARK-2954] [SPARK-2948] [SPARK-2910] [SPARK-2101] Python 2.6 Fixes - Modify python/run-tests to test with Python 2.6 - Use unittest2 when running on Python 2.6. - Fix issue with namedtuple. - Skip TestOutputFormat.test_newhadoop on Python 2.6 until SPARK-2951 is fixed. - Fix MLlib _deserialize_double on Python 2.6. [SPARK-3867][PySpark] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed ./python/run-tests search a Python 2.6 executable on PATH and use it if available. When using Python 2.6, it is going to import unittest2 module which is not a standard library in Python 2.6, so it fails with Import Author: cocoatomo Author: Josh Rosen Author: Davies Liu Author: Davies Liu Closes #3668 from davies/port_2365 and squashes the following commits: b32583d [Davies Liu] rollback _common.py bda1c72 [cocoatomo] [SPARK-3867][PySpark] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed 14ad3d9 [Josh Rosen] [PySpark] [SPARK-2954] [SPARK-2948] [SPARK-2910] [SPARK-2101] Python 2.6 Fixes 7c55cff [Davies Liu] [SPARK-2951] [PySpark] support unpickle array.array for Python 2.6 Conflicts: python/pyspark/mllib/tests.py python/pyspark/tests.py python/run-tests --- python/pyspark/tests.py | 11 ++++++++++- python/run-tests | 10 ++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/python/pyspark/tests.py b/python/pyspark/tests.py index 5b124d9308562..81ef80e713eee 100644 --- a/python/pyspark/tests.py +++ b/python/pyspark/tests.py @@ -26,7 +26,16 @@ import sys from tempfile import NamedTemporaryFile import time -import unittest + +if sys.version_info[:2] <= (2, 6): + try: + import unittest2 as unittest + except ImportError: + sys.stderr.write('Please install unittest2 to test with Python 2.6 or earlier') + sys.exit(1) +else: + import unittest + from pyspark.context import SparkContext from pyspark.files import SparkFiles diff --git a/python/run-tests b/python/run-tests index a986ac9380be4..6eeef4c13680e 100755 --- a/python/run-tests +++ b/python/run-tests @@ -33,6 +33,16 @@ function run_test() { FAILED=$((PIPESTATUS[0]||$FAILED)) } +echo "Running PySpark tests. Output is in python/unit-tests.log." + +# Try to test with Python 2.6, since that's the minimum version that we support: +if [ $(which python2.6) ]; then + export PYSPARK_PYTHON="python2.6" +fi + +echo "Testing with Python version:" +$PYSPARK_PYTHON --version + run_test "pyspark/rdd.py" run_test "pyspark/context.py" run_test "pyspark/conf.py"