Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-24571][SQL] Support Char literals #21578

Closed
wants to merge 10 commits into from

Conversation

MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Jun 16, 2018

What changes were proposed in this pull request?

In the PR, I propose to automatically convert a Literal with Char type to a Literal of String type. Currently, the following code:

val df = Seq("Amsterdam", "San Francisco", "London").toDF("city")
df.where($"city".contains('o')).show(false)

fails with the exception:

Unsupported literal type class java.lang.Character o
java.lang.RuntimeException: Unsupported literal type class java.lang.Character o
at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:78)

The PR fixes this issue by converting char to string of length 1. I believe it makes sense to does not differentiate char and string(1) in a unified, multi-language data platform like Spark which supports languages like Python/R.

@SparkQA
Copy link

SparkQA commented Jun 16, 2018

Test build #91961 has finished for PR 21578 at commit 4210146.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MaxGekk
Copy link
Member Author

MaxGekk commented Jun 16, 2018

jenkins, retest this, please

@SparkQA
Copy link

SparkQA commented Jun 16, 2018

Test build #91974 has finished for PR 21578 at commit 4210146.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MaxGekk
Copy link
Member Author

MaxGekk commented Jun 16, 2018

@rxin Please, review the PR.

Copy link
Member

@gatorsmile gatorsmile left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gatorsmile
Copy link
Member

@MaxGekk Could you resolve the conflict and then I will merge it?

@SparkQA
Copy link

SparkQA commented Jun 19, 2018

Test build #92099 has finished for PR 21578 at commit 450079d.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 19, 2018

Test build #92100 has finished for PR 21578 at commit 3926655.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

@gengliangwang gengliangwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gatorsmile
Copy link
Member

Thanks! Merged to master.

@asfgit asfgit closed this in 54fcaaf Jun 21, 2018
@MaxGekk MaxGekk deleted the support-char-literals branch August 17, 2019 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants