[SPARK-18220][SQL] read Hive orc table with varchar column should not fail #16060
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #69325 has finished for PR 16060 at commit
|
cloud-fan
changed the title
|
Test build #69327 has finished for PR 16060 at commit
|
rxin
reviewed
Nov 29, 2016
| @@ -51,9 +51,12 @@ private[spark] object HiveUtils extends Logging { | |||
| sc | |||
| } | |||
|
|
|||
| /** The version of hive used internally by Spark SQL. */ | |||
| // The version of hive used internally by Spark SQL. | |||
There was a problem hiding this comment.
why did you change the comment style?
rxin
reviewed
Nov 29, 2016
| val hiveExecutionVersion: String = "1.2.1" | ||
|
|
||
| // The property key that is used to store the raw hive type string in the metadata of StructField. |
There was a problem hiding this comment.
I'd add a bit more color here, e.g. by adding an example: "For example, in the case where the Hive type is varchar, the type gets mapped to a string type in Spark SQL, but we need to preserve the original type in order to invoke the correct object inspector in Hive"
|
Test build #69389 has started for PR 16060 at commit |
|
Test build #3446 has finished for PR 16060 at commit
|
|
Merging in master/branch-2.1. |
asfgit
pushed a commit
that referenced
this pull request
Nov 30, 2016
… fail ## What changes were proposed in this pull request? Spark SQL only has `StringType`, when reading hive table with varchar column, we will read that column as `StringType`. However, we still need to use varchar `ObjectInspector` to read varchar column in hive table, which means we need to know the actual column type at hive side. In Spark 2.1, after #14363 , we parse hive type string to catalyst type, which means the actual column type at hive side is erased. Then we may use string `ObjectInspector` to read varchar column and fail. This PR keeps the original hive column type string in the metadata of `StructField`, and use it when we convert it to a hive column. ## How was this patch tested? newly added regression test Author: Wenchen Fan <[email protected]> Closes #16060 from cloud-fan/varchar. (cherry picked from commit 3f03c90) Signed-off-by: Reynold Xin <[email protected]>
robert3005
pushed a commit
to palantir/spark
that referenced
this pull request
Dec 2, 2016
… fail ## What changes were proposed in this pull request? Spark SQL only has `StringType`, when reading hive table with varchar column, we will read that column as `StringType`. However, we still need to use varchar `ObjectInspector` to read varchar column in hive table, which means we need to know the actual column type at hive side. In Spark 2.1, after apache#14363 , we parse hive type string to catalyst type, which means the actual column type at hive side is erased. Then we may use string `ObjectInspector` to read varchar column and fail. This PR keeps the original hive column type string in the metadata of `StructField`, and use it when we convert it to a hive column. ## How was this patch tested? newly added regression test Author: Wenchen Fan <[email protected]> Closes apache#16060 from cloud-fan/varchar.
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
… fail ## What changes were proposed in this pull request? Spark SQL only has `StringType`, when reading hive table with varchar column, we will read that column as `StringType`. However, we still need to use varchar `ObjectInspector` to read varchar column in hive table, which means we need to know the actual column type at hive side. In Spark 2.1, after apache#14363 , we parse hive type string to catalyst type, which means the actual column type at hive side is erased. Then we may use string `ObjectInspector` to read varchar column and fail. This PR keeps the original hive column type string in the metadata of `StructField`, and use it when we convert it to a hive column. ## How was this patch tested? newly added regression test Author: Wenchen Fan <[email protected]> Closes apache#16060 from cloud-fan/varchar.
asfgit
pushed a commit
that referenced
this pull request
Feb 10, 2017
…tadata ## What changes were proposed in this pull request? Reading from an existing ORC table which contains `char` or `varchar` columns can fail with a `ClassCastException` if the table metadata has been created using Spark. This is caused by the fact that spark internally replaces `char` and `varchar` columns with a `string` column. This PR fixes this by adding the hive type to the `StructField's` metadata under the `HIVE_TYPE_STRING` key. This is picked up by the `HiveClient` and the ORC reader, see #16060 for more details on how the metadata is used. ## How was this patch tested? Added a regression test to `OrcSourceSuite`. Author: Herman van Hovell <[email protected]> Closes #16804 from hvanhovell/SPARK-19459.
cmonkey
pushed a commit
to cmonkey/spark
that referenced
this pull request
Feb 15, 2017
…tadata ## What changes were proposed in this pull request? Reading from an existing ORC table which contains `char` or `varchar` columns can fail with a `ClassCastException` if the table metadata has been created using Spark. This is caused by the fact that spark internally replaces `char` and `varchar` columns with a `string` column. This PR fixes this by adding the hive type to the `StructField's` metadata under the `HIVE_TYPE_STRING` key. This is picked up by the `HiveClient` and the ORC reader, see apache#16060 for more details on how the metadata is used. ## How was this patch tested? Added a regression test to `OrcSourceSuite`. Author: Herman van Hovell <[email protected]> Closes apache#16804 from hvanhovell/SPARK-19459.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Spark SQL only has
StringType, when reading hive table with varchar column, we will read that column asStringType. However, we still need to use varcharObjectInspectorto read varchar column in hive table, which means we need to know the actual column type at hive side.In Spark 2.1, after #14363 , we parse hive type string to catalyst type, which means the actual column type at hive side is erased. Then we may use string
ObjectInspectorto read varchar column and fail.This PR keeps the original hive column type string in the metadata of
StructField, and use it when we convert it to a hive column.How was this patch tested?
newly added regression test