[SPARK-6661] Python type errors should print type, not object

[31z4]

No description provided.

[rxin]

Jenkins, test this please.

[rxin]

cc @davies

[davies]

@31z4 Thanks to look into this. Personally, I like the type object instead of name of type, for example, you can got the full name of a class:

>>> print "but got:", type("")
but got: <type 'str'>
>>> print "but got:", type("").__name__
but got: str
>>> print pyspark.RDD
<class 'pyspark.rdd.RDD'>
>>> print pyspark.RDD.__name__
RDD

Also, we may change some ValueError to TypeError.

[31z4]

@davies Thanks for your comments! I made changes.

[davies]

@31z4 After reading the docs again, it should be TypeError, see https://docs.python.org/2/reference/datamodel.html#object.__getitem__

>>> ""[""]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: string indices must be integers, not str

Sorry for that.

[31z4]

@davies It's OK. I may have looked into docs by myself :)

[SparkQA]

Test build #639 has finished for PR 5361 at commit c4385aa.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[davies]

@31z4 You can run dev/lint-python to find the lint errors.

[31z4]

@davies Fixed and rebased.

[SparkQA]

Test build #641 has finished for PR 5361 at commit 619ffd3.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[31z4]

@davies Fixed errors and rebased.

[SparkQA]

Test build #653 has finished for PR 5361 at commit 650ad45.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[31z4]

@davies seems like the failures are in Scala tests. Can anyone look into that? I also had failures in Scala tests on my machine. All Python tests passed without errors though.

[SparkQA]

Test build #654 has finished for PR 5361 at commit 650ad45.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #658 has finished for PR 5361 at commit 650ad45.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch removes the following dependencies:
  • RoaringBitmap-0.4.5.jar
  • activation-1.1.jar
  • akka-actor_2.10-2.3.4-spark.jar
  • akka-remote_2.10-2.3.4-spark.jar
  • akka-slf4j_2.10-2.3.4-spark.jar
  • aopalliance-1.0.jar
  • arpack_combined_all-0.1.jar
  • avro-1.7.7.jar
  • breeze-macros_2.10-0.11.2.jar
  • breeze_2.10-0.11.2.jar
  • chill-java-0.5.0.jar
  • chill_2.10-0.5.0.jar
  • commons-beanutils-1.7.0.jar
  • commons-beanutils-core-1.8.0.jar
  • commons-cli-1.2.jar
  • commons-codec-1.10.jar
  • commons-collections-3.2.1.jar
  • commons-compress-1.4.1.jar
  • commons-configuration-1.6.jar
  • commons-digester-1.8.jar
  • commons-httpclient-3.1.jar
  • commons-io-2.1.jar
  • commons-lang-2.5.jar
  • commons-lang3-3.3.2.jar
  • commons-math-2.1.jar
  • commons-math3-3.1.1.jar
  • commons-net-2.2.jar
  • compress-lzf-1.0.0.jar
  • config-1.2.1.jar
  • core-1.1.2.jar
  • curator-client-2.4.0.jar
  • curator-framework-2.4.0.jar
  • curator-recipes-2.4.0.jar
  • gmbal-api-only-3.0.0-b023.jar
  • grizzly-framework-2.1.2.jar
  • grizzly-http-2.1.2.jar
  • grizzly-http-server-2.1.2.jar
  • grizzly-http-servlet-2.1.2.jar
  • grizzly-rcm-2.1.2.jar
  • groovy-all-2.3.7.jar
  • guava-14.0.1.jar
  • guice-3.0.jar
  • hadoop-annotations-2.2.0.jar
  • hadoop-auth-2.2.0.jar
  • hadoop-client-2.2.0.jar
  • hadoop-common-2.2.0.jar
  • hadoop-hdfs-2.2.0.jar
  • hadoop-mapreduce-client-app-2.2.0.jar
  • hadoop-mapreduce-client-common-2.2.0.jar
  • hadoop-mapreduce-client-core-2.2.0.jar
  • hadoop-mapreduce-client-jobclient-2.2.0.jar
  • hadoop-mapreduce-client-shuffle-2.2.0.jar
  • hadoop-yarn-api-2.2.0.jar
  • hadoop-yarn-client-2.2.0.jar
  • hadoop-yarn-common-2.2.0.jar
  • hadoop-yarn-server-common-2.2.0.jar
  • ivy-2.4.0.jar
  • jackson-annotations-2.4.0.jar
  • jackson-core-2.4.4.jar
  • jackson-core-asl-1.8.8.jar
  • jackson-databind-2.4.4.jar
  • jackson-jaxrs-1.8.8.jar
  • jackson-mapper-asl-1.8.8.jar
  • jackson-module-scala_2.10-2.4.4.jar
  • jackson-xc-1.8.8.jar
  • jansi-1.4.jar
  • javax.inject-1.jar
  • javax.servlet-3.0.0.v201112011016.jar
  • javax.servlet-3.1.jar
  • javax.servlet-api-3.0.1.jar
  • jaxb-api-2.2.2.jar
  • jaxb-impl-2.2.3-1.jar
  • jcl-over-slf4j-1.7.10.jar
  • jersey-client-1.9.jar
  • jersey-core-1.9.jar
  • jersey-grizzly2-1.9.jar
  • jersey-guice-1.9.jar
  • jersey-json-1.9.jar
  • jersey-server-1.9.jar
  • jersey-test-framework-core-1.9.jar
  • jersey-test-framework-grizzly2-1.9.jar
  • jets3t-0.7.1.jar
  • jettison-1.1.jar
  • jetty-util-6.1.26.jar
  • jline-0.9.94.jar
  • jline-2.10.4.jar
  • jodd-core-3.6.3.jar
  • json4s-ast_2.10-3.2.10.jar
  • json4s-core_2.10-3.2.10.jar
  • json4s-jackson_2.10-3.2.10.jar
  • jsr305-1.3.9.jar
  • jtransforms-2.4.0.jar
  • jul-to-slf4j-1.7.10.jar
  • kryo-2.21.jar
  • log4j-1.2.17.jar
  • lz4-1.2.0.jar
  • management-api-3.0.0-b012.jar
  • mesos-0.21.0-shaded-protobuf.jar
  • metrics-core-3.1.0.jar
  • metrics-graphite-3.1.0.jar
  • metrics-json-3.1.0.jar
  • metrics-jvm-3.1.0.jar
  • minlog-1.2.jar
  • netty-3.8.0.Final.jar
  • netty-all-4.0.23.Final.jar
  • objenesis-1.2.jar
  • opencsv-2.3.jar
  • oro-2.0.8.jar
  • paranamer-2.6.jar
  • parquet-column-1.6.0rc3.jar
  • parquet-common-1.6.0rc3.jar
  • parquet-encoding-1.6.0rc3.jar
  • parquet-format-2.2.0-rc1.jar
  • parquet-generator-1.6.0rc3.jar
  • parquet-hadoop-1.6.0rc3.jar
  • parquet-jackson-1.6.0rc3.jar
  • protobuf-java-2.4.1.jar
  • protobuf-java-2.5.0-spark.jar
  • py4j-0.8.2.1.jar
  • pyrolite-2.0.1.jar
  • quasiquotes_2.10-2.0.1.jar
  • reflectasm-1.07-shaded.jar
  • scala-compiler-2.10.4.jar
  • scala-library-2.10.4.jar
  • scala-reflect-2.10.4.jar
  • scalap-2.10.4.jar
  • scalatest_2.10-2.2.1.jar
  • slf4j-api-1.7.10.jar
  • slf4j-log4j12-1.7.10.jar
  • snappy-java-1.1.1.6.jar
  • spark-bagel_2.10-1.4.0-SNAPSHOT.jar
  • spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
  • spark-core_2.10-1.4.0-SNAPSHOT.jar
  • spark-graphx_2.10-1.4.0-SNAPSHOT.jar
  • spark-launcher_2.10-1.4.0-SNAPSHOT.jar
  • spark-mllib_2.10-1.4.0-SNAPSHOT.jar
  • spark-network-common_2.10-1.4.0-SNAPSHOT.jar
  • spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
  • spark-repl_2.10-1.4.0-SNAPSHOT.jar
  • spark-sql_2.10-1.4.0-SNAPSHOT.jar
  • spark-streaming_2.10-1.4.0-SNAPSHOT.jar
  • spire-macros_2.10-0.7.4.jar
  • spire_2.10-0.7.4.jar
  • stax-api-1.0.1.jar
  • stream-2.7.0.jar
  • tachyon-0.5.0.jar
  • tachyon-client-0.5.0.jar
  • uncommons-maths-1.2.2a.jar
  • unused-1.0.0.jar
  • xmlenc-0.52.jar
  • xz-1.0.jar
  • zookeeper-3.4.5.jar

[31z4]

@davies What should I do? Either just wait when master will be fixed or rebase to an exact commit?

[jkbradley]

@davies I agree it's nice to get the actual object printed, rather than just the type. My problem with it was it failing to print meaningful error messages when the object did not have a default conversion to a string. (It would then print an error about the conversion, rather than about the original error.)

@31z4 What if we kept printing the object (not the type) as before, but did it via a helper method used as follows:

  raise TypeError("Found an error with: " + tryToString(myObject))

where tryToString first tries printing the object to a string ("object blah") and, if that fails, then tries printing the type to a string ("type: objecttype")?

[jkbradley]

@davies Oops, I misinterpreted your above comment. If we're sticking with types, then nevermind!

[31z4]

@jkbradley Yes, we're sticking with types.

[31z4]

@davies @jkbradley @rxin could someone look into this commit again and trigger the Jenkins test?

[SparkQA]

Test build #677 has started for PR 5361 at commit 228a0f8.

[JoshRosen]

It looks like this latest test run actually passed, but there was a problem in posting the success message to GitHub. I haven't been following this PR, but if other committers think it's good to go then we might want to merge it now so that it doesn't merge conflict when the Python 3 patch is merged.

[31z4]

Thanks @JoshRosen! Looking forward for someone's answer.

[davies]

LGTM

[mengxr]

test this please

[SparkQA]

Test build #30403 has finished for PR 5361 at commit 228a0f8.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds no public classes.
• This patch adds the following new dependencies:
  • commons-math3-3.1.1.jar
  • snappy-java-1.1.1.6.jar
• This patch removes the following dependencies:
  • commons-math3-3.4.1.jar
  • snappy-java-1.1.1.7.jar

[JoshRosen]

It looks like this is now merge-conflicted with master due to the Python 3 change, but I think that I should be able to resolve the conflicts myself on merge.

[JoshRosen]

Actually, it looks like the MLlib conflicts may not be all due to the Python 3 change, so do you mind fixing up these conflicts? Sorry for not preventing these conflicts by merging earlier.

[31z4]

@JoshRosen I don't mind. I'll do it.

[31z4]

@JoshRosen Done.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #30576 has finished for PR 5361 at commit 73c5d79.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[JoshRosen]

LGTM, so I'm going to merge this into master (1.4.0). Thanks!