[SPARK-7295][SQL] bitwise operations for DataFrame DSL #5867

Shiti · 2015-05-02T21:06:01Z

No description provided.

AmplabJenkins · 2015-05-02T21:07:09Z

Merged build triggered.

AmplabJenkins · 2015-05-02T21:07:19Z

Merged build started.

SparkQA · 2015-05-02T21:08:49Z

Test build #31669 has started for PR 5867 at commit f08e595.

SparkQA · 2015-05-02T23:18:27Z

Test build #31669 has finished for PR 5867 at commit f08e595.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-02T23:18:32Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-02T23:18:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31669/
Test PASSed.

rxin · 2015-05-03T04:53:32Z

python/pyspark/sql/dataframe.py

@@ -1214,6 +1214,12 @@ def __init__(self, jc):
    __contains__ = _bin_op("contains")
    __getitem__ = _bin_op("getItem")

+    # bitwise operators
+    __bitwiseor__ = _bin_op("bitwiseOR")


i don't think this means anything in Python. You can just do

bitwiseOR = _bin_op("bitwiseOR") bitwiseAND = _bin_op("bitwiseOR") bitwiseXOR = _bin_op("bitwiseOR") bitwiseNOT = _bin_op("bitwiseOR")

AmplabJenkins · 2015-05-03T06:32:10Z

Merged build triggered.

AmplabJenkins · 2015-05-03T06:32:17Z

Merged build started.

SparkQA · 2015-05-03T06:33:51Z

Test build #31682 has started for PR 5867 at commit 72e26d1.

SparkQA · 2015-05-03T08:47:50Z

Test build #31682 has finished for PR 5867 at commit 72e26d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-03T08:47:55Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-03T08:47:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31682/
Test PASSed.

rxin · 2015-05-03T19:43:40Z

sql/core/src/main/scala/org/apache/spark/sql/Column.scala

+
+  def bitwiseXOR(other: Any): Column = BitwiseXor(expr, lit(other).expr)
+
+  def bitwiseNOT : Column = BitwiseNot(expr)


sorry after looking at it more, I have a new round of comments.

we should move bitwiseNOT into functions.py

AmplabJenkins · 2015-05-03T20:02:11Z

Merged build triggered.

AmplabJenkins · 2015-05-03T20:02:21Z

Merged build started.

AmplabJenkins · 2015-05-03T20:07:09Z

Merged build triggered.

AmplabJenkins · 2015-05-03T20:07:16Z

Merged build started.

SparkQA · 2015-05-03T20:08:50Z

Test build #31709 has started for PR 5867 at commit 72126a6.

SparkQA · 2015-05-03T20:10:21Z

Test build #31709 has finished for PR 5867 at commit 72126a6.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-03T20:10:22Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-03T20:10:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31709/
Test FAILed.

AmplabJenkins · 2015-05-03T20:17:10Z

Merged build triggered.

AmplabJenkins · 2015-05-03T20:17:16Z

Merged build started.

AmplabJenkins · 2015-05-03T20:17:24Z

Merged build finished. Test FAILed.

SparkQA · 2015-05-06T19:02:54Z

Test build #32016 has started for PR 5867 at commit b4b3e67.

SparkQA · 2015-05-06T21:02:00Z

Test build #32016 has finished for PR 5867 at commit b4b3e67.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-06T21:02:06Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-06T21:02:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32016/
Test FAILed.

Shiti · 2015-05-07T02:14:06Z

@rxin All tests are passed when I run them locally but in Jenkins build, one of them failed.

File "pyspark/ml/tuning.py", line 108, in __main__.CrossValidator
Failed example:
    cvModel.transform(dataset).collect() == expected.collect()
Expected:
    True
Got:
    False

How can I fix this?

rxin · 2015-05-07T05:19:52Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

@@ -438,6 +438,9 @@ object functions {
   */
  def upper(e: Column): Column = Upper(e.expr)

+
+  def bitwiseNOT(e: Column) : Column = BitwiseNot(e.expr)


Add scaladoc, and remove the space before :

rxin · 2015-05-07T05:20:48Z

@jkbradley is there some non-determinism with the Python test?

jkbradley · 2015-05-07T05:35:50Z

It's been removed and is being investigated: [https://issues.apache.org/jira/browse/SPARK-7432]

jkbradley · 2015-05-07T05:36:27Z

Jenkins test this please

AmplabJenkins · 2015-05-07T05:37:10Z

Merged build triggered.

AmplabJenkins · 2015-05-07T05:37:19Z

Merged build started.

SparkQA · 2015-05-07T05:37:32Z

Test build #32078 has started for PR 5867 at commit b4b3e67.

…nd docs

AmplabJenkins · 2015-05-07T06:07:10Z

Merged build triggered.

AmplabJenkins · 2015-05-07T06:07:19Z

Merged build started.

SparkQA · 2015-05-07T06:09:08Z

Test build #32081 has started for PR 5867 at commit 71a9913.

SparkQA · 2015-05-07T07:56:25Z

Test build #32078 has finished for PR 5867 at commit b4b3e67.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-07T07:56:29Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-07T07:56:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32078/
Test PASSed.

rxin · 2015-05-07T08:00:11Z

Thanks. I'm merging this in.

Author: Shiti <[email protected]> Closes #5867 from Shiti/spark-7295 and squashes the following commits: 71a9913 [Shiti] implementation for bitwise and,or, not and xor on Column with tests and docs (cherry picked from commit fa8fddf) Signed-off-by: Reynold Xin <[email protected]>

SparkQA · 2015-05-07T08:39:09Z

Test build #32081 timed out for PR 5867 at commit 71a9913 after a configured wait of 150m.

AmplabJenkins · 2015-05-07T08:39:13Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-07T08:39:13Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32081/
Test FAILed.

Author: Shiti <[email protected]> Closes apache#5867 from Shiti/spark-7295 and squashes the following commits: 71a9913 [Shiti] implementation for bitwise and,or, not and xor on Column with tests and docs

Shiti changed the title ~~[SPARK-7294][SQL] bitwise operations for DataFrame DSL~~ [SPARK-7295][SQL] bitwise operations for DataFrame DSL May 2, 2015

rxin reviewed May 3, 2015
View reviewed changes

rxin reviewed May 7, 2015
View reviewed changes

implementation for bitwise and,or, not and xor on Column with tests a…

71a9913

…nd docs

Shiti force-pushed the spark-7295 branch from b4b3e67 to 71a9913 Compare May 7, 2015 06:04

asfgit closed this in fa8fddf May 7, 2015

srowen mentioned this pull request May 30, 2018

[SPARK-24408][SQL][DOC] Move abs function to math_funcs group #21448

Closed


		def bitwiseXOR(other: Any): Column = BitwiseXor(expr, lit(other).expr)

		def bitwiseNOT : Column = BitwiseNot(expr)

[SPARK-7295][SQL] bitwise operations for DataFrame DSL #5867

[SPARK-7295][SQL] bitwise operations for DataFrame DSL #5867

Conversation

Shiti commented May 2, 2015

AmplabJenkins commented May 2, 2015

AmplabJenkins commented May 2, 2015

SparkQA commented May 2, 2015

SparkQA commented May 2, 2015

AmplabJenkins commented May 2, 2015

AmplabJenkins commented May 2, 2015

rxin May 3, 2015

Choose a reason for hiding this comment

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

SparkQA commented May 3, 2015

SparkQA commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

rxin May 3, 2015

Choose a reason for hiding this comment

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

SparkQA commented May 3, 2015

SparkQA commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

SparkQA commented May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

Shiti commented May 7, 2015

rxin May 7, 2015

Choose a reason for hiding this comment

rxin commented May 7, 2015

jkbradley commented May 7, 2015

jkbradley commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

rxin commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015