[SPARK-7321][SQL] Add Column expression for conditional statements (when/otherwise) #6072

rxin · 2015-05-12T05:26:54Z

This builds on #5932 and should close #5932 as well.

As an example:

df.select(when(df['age'] == 2, 3).otherwise(4).alias("age")).collect()

…f, case)

[SPARK-7321][SQL] Add Column expression for conditional statements

AmplabJenkins · 2015-05-12T05:27:10Z

Merged build triggered.

AmplabJenkins · 2015-05-12T05:27:18Z

Merged build started.

AmplabJenkins · 2015-05-12T05:32:10Z

Merged build triggered.

AmplabJenkins · 2015-05-12T05:32:18Z

Merged build started.

SparkQA · 2015-05-12T05:33:12Z

Test build #32464 has started for PR 6072 at commit 0455eda.

AmplabJenkins · 2015-05-12T05:42:22Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-12T05:42:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32463/
Test FAILed.

rxin · 2015-05-12T05:43:51Z

Jenkins, retest this please.

AmplabJenkins · 2015-05-12T05:47:10Z

Merged build triggered.

AmplabJenkins · 2015-05-12T05:47:18Z

Merged build started.

SparkQA · 2015-05-12T05:49:07Z

Test build #32466 has started for PR 6072 at commit 0455eda.

cloud-fan · 2015-05-12T07:04:31Z

How about "case key when" syntax? Like "CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END" in SQL and choose(df("key")).when("a", 1).when("b", 2).else(3).

rxin · 2015-05-12T07:09:34Z

the one you suggested is more like switch function in C/Java. The one here is just general if/else. We should add that one too, but in separate PR.

SparkQA · 2015-05-12T07:48:08Z

Test build #32464 has finished for PR 6072 at commit 0455eda.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-12T07:48:13Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-12T07:48:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32464/
Test PASSed.

SparkQA · 2015-05-12T08:12:45Z

Test build #32466 has finished for PR 6072 at commit 0455eda.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-12T08:12:51Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-12T08:12:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32466/
Test PASSed.

cloud-fan · 2015-05-12T08:49:33Z

sql/core/src/main/scala/org/apache/spark/sql/Column.scala

+   */
+  def otherwise(value: Any):Column = this.expr match {
+    case CaseWhen(branches: Seq[Expression]) =>
+      CaseWhen(branches :+ lit(value).expr)


What if user mistakenly call otherwise twice? Then we will build a wrong CaseWhen expression here. Maybe we should create a helper class like what we did for NA functions and aggregate functions?

Good point. It is too heavy weight to have a whole class dedicated to this, but I will add code to throw exceptions if otherwise has been applied previously.

AmplabJenkins · 2015-05-12T19:12:11Z

Merged build triggered.

AmplabJenkins · 2015-05-12T19:12:19Z

Merged build started.

SparkQA · 2015-05-12T19:14:09Z

Test build #32525 has started for PR 6072 at commit 8f49201.

SparkQA · 2015-05-12T21:44:10Z

Test build #32525 timed out for PR 6072 at commit 8f49201 after a configured wait of 150m.

AmplabJenkins · 2015-05-12T21:44:15Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-12T21:44:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32525/
Test FAILed.

rxin · 2015-05-12T22:06:14Z

Jenkins, retest this please.

AmplabJenkins · 2015-05-12T22:07:11Z

Merged build triggered.

AmplabJenkins · 2015-05-12T22:07:20Z

Merged build started.

SparkQA · 2015-05-12T22:07:43Z

Test build #32542 has started for PR 6072 at commit 8f49201.

SparkQA · 2015-05-13T00:34:47Z

Test build #32542 has finished for PR 6072 at commit 8f49201.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-13T00:34:56Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-13T00:34:56Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32542/
Test FAILed.

rxin · 2015-05-13T00:38:13Z

Jenkins, retest this please.

AmplabJenkins · 2015-05-13T00:42:12Z

Merged build triggered.

AmplabJenkins · 2015-05-13T00:42:22Z

Merged build started.

SparkQA · 2015-05-13T00:42:37Z

Test build #32557 has started for PR 6072 at commit 8f49201.

SparkQA · 2015-05-13T03:07:14Z

Test build #32557 has finished for PR 6072 at commit 8f49201.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-13T03:07:19Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-13T03:07:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32557/
Test PASSed.

…hen/otherwise) This builds on #5932 and should close #5932 as well. As an example: ```python df.select(when(df['age'] == 2, 3).otherwise(4).alias("age")).collect() ``` Author: Reynold Xin <[email protected]> Author: kaka1992 <[email protected]> Closes #6072 from rxin/when-expr and squashes the following commits: 8f49201 [Reynold Xin] Throw exception if otherwise is applied twice. 0455eda [Reynold Xin] Reset run-tests. bfb9d9f [Reynold Xin] Updated documentation and test cases. 762f6a5 [Reynold Xin] Merge pull request #5932 from kaka1992/IFCASE 95724c6 [kaka1992] Update 8218d0a [kaka1992] Update 801009e [kaka1992] Update 76d6346 [kaka1992] [SPARK-7321][SQL] Add Column expression for conditional statements (if, case) (cherry picked from commit 97dee31) Signed-off-by: Reynold Xin <[email protected]>

…hen/otherwise) This builds on apache#5932 and should close apache#5932 as well. As an example: ```python df.select(when(df['age'] == 2, 3).otherwise(4).alias("age")).collect() ``` Author: Reynold Xin <[email protected]> Author: kaka1992 <[email protected]> Closes apache#6072 from rxin/when-expr and squashes the following commits: 8f49201 [Reynold Xin] Throw exception if otherwise is applied twice. 0455eda [Reynold Xin] Reset run-tests. bfb9d9f [Reynold Xin] Updated documentation and test cases. 762f6a5 [Reynold Xin] Merge pull request apache#5932 from kaka1992/IFCASE 95724c6 [kaka1992] Update 8218d0a [kaka1992] Update 801009e [kaka1992] Update 76d6346 [kaka1992] [SPARK-7321][SQL] Add Column expression for conditional statements (if, case)

kaka1992 and others added 6 commits May 6, 2015 11:20

[SPARK-7321][SQL] Add Column expression for conditional statements (i…

76d6346

…f, case)

Update

801009e

Update

8218d0a

Update

95724c6

Merge pull request apache#5932 from kaka1992/IFCASE

762f6a5

[SPARK-7321][SQL] Add Column expression for conditional statements

Updated documentation and test cases.

bfb9d9f

Reset run-tests.

0455eda

rxin mentioned this pull request May 12, 2015

[SPARK-7321][SQL] Add Column expression for conditional statements #5932

Closed

cloud-fan reviewed May 12, 2015
View reviewed changes

Throw exception if otherwise is applied twice.

8f49201

asfgit closed this in 97dee31 May 13, 2015

[SPARK-7321][SQL] Add Column expression for conditional statements (when/otherwise) #6072

[SPARK-7321][SQL] Add Column expression for conditional statements (when/otherwise) #6072

Conversation

rxin commented May 12, 2015

AmplabJenkins commented May 12, 2015

AmplabJenkins commented May 12, 2015

AmplabJenkins commented May 12, 2015

AmplabJenkins commented May 12, 2015

SparkQA commented May 12, 2015

AmplabJenkins commented May 12, 2015

AmplabJenkins commented May 12, 2015

rxin commented May 12, 2015

AmplabJenkins commented May 12, 2015

AmplabJenkins commented May 12, 2015

SparkQA commented May 12, 2015

cloud-fan commented May 12, 2015

rxin commented May 12, 2015

SparkQA commented May 12, 2015

AmplabJenkins commented May 12, 2015

AmplabJenkins commented May 12, 2015

SparkQA commented May 12, 2015

AmplabJenkins commented May 12, 2015

AmplabJenkins commented May 12, 2015

cloud-fan May 12, 2015

Choose a reason for hiding this comment

rxin May 12, 2015

Choose a reason for hiding this comment

AmplabJenkins commented May 12, 2015

AmplabJenkins commented May 12, 2015

SparkQA commented May 12, 2015

SparkQA commented May 12, 2015

AmplabJenkins commented May 12, 2015

AmplabJenkins commented May 12, 2015

rxin commented May 12, 2015

AmplabJenkins commented May 12, 2015

AmplabJenkins commented May 12, 2015

SparkQA commented May 12, 2015

SparkQA commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

rxin commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

SparkQA commented May 13, 2015

SparkQA commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015