[SPARK-37507][SQL] Add a new SQL function to_binary #35415

xinrong-meng · 2022-02-07T07:33:04Z

What changes were proposed in this pull request?

Introduce a SQL function to_binary: Converts the input string to a binary value based on the supplied format (of how to interpret the string).

Syntax:

to_binary(str_column[, fmt])

where

fmt can be a case-insensitive string literal of "hex", "utf-8", "base2", or "base64".
By default, the binary format for conversion is "hex" if fmt is omitted.

Why are the changes needed?

to_binary is a common function available in many DBMSes, for example:

Introducing it improves compatibility and the ease of migration.

In addition, to_binary can unify existing Spark functions: encode, unhex, unbase64, and binary, which makes API easier to remember and use.

Does this PR introduce any user-facing change?

Yes, a new function for the string to binary conversion with a specified format.

How was this patch tested?

Unit test.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala

HyukjinKwon · 2022-02-10T04:13:37Z

Looks fine from a cursory look. cc @cloud-fan, @MaxGekk and @gengliangwang in case you guys find some time to review.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala

xinrong-meng · 2022-02-10T07:07:13Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala

+@ExpressionDescription(
+  usage = """
+    _FUNC_(str[, fmt]) - Converts the input `str` to a binary value based on the supplied `fmt`.
+      By default, the binary format for conversion is "hex" if `fmt` is omitted.


Setting hex as the default format referencing reference.

sql/core/src/test/resources/sql-tests/results/ansi/string-functions.sql.out

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala

xinrong-meng · 2022-02-11T05:29:39Z

sql/core/src/test/resources/sql-tests/results/string-functions.sql.out

+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'to_binary('abc', 'invalidFormat')' due to data type mismatch: Unsupported encoding format: Some(invalidFormat). The format has to be a case-insensitive string literal of 'hex', 'utf-8', 'base2', or 'base64'; line 1 pos 7


Intentionally failed query for testing.

sql/core/src/test/resources/sql-functions/sql-expression-schema.md

xinrong-meng · 2022-02-11T12:56:01Z

Sorry I rebased the PR so the commit history looks confusing, but only two commits below are new:

300f1e8
3fd1b77

CC @HyukjinKwon

HyukjinKwon · 2022-02-13T00:32:19Z

Merged to master.

cloud-fan · 2022-02-14T03:16:20Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala

+        val value = lit.eval()
+        if (value == null) Literal(null, BinaryType)
+        else {
+          value.asInstanceOf[UTF8String].toString.toLowerCase(Locale.ROOT) match {


hmm, shall we check the type of format first? What happens to to_binary('abc', 1)?

Added the input check in #35533.

entong · 2022-02-14T07:54:33Z

sql/core/src/test/resources/sql-tests/results/ansi/string-functions.sql.out

+
+
+-- !query
+select to_binary('abc', 'base2')


Should this be an error instead? For base2 it is expecting a string of 0s and 1s.

Please correct me if I'm wrong:

My understanding is base2(binary) should use the default encoding/decoding format, it happens to be utf-8 here.

We may also exclude base2, following what Snowflake to_binary does.

Also CC @cloud-fan @HyukjinKwon @gengliangwang

let's exclude base2 as its behavior is a bit arguable.

cloud-fan · 2022-02-15T02:26:08Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala

+            case "hex" => Unhex(expr)
+            case "utf-8" => Encode(expr, Literal("UTF-8"))
+            case "base64" => UnBase64(expr)
+            case "base2" => Cast(expr, BinaryType)


I think this is the same as Encode(expr, Literal("UTF-8")), not sure it works for base2

xinrong-meng · 2022-02-15T05:27:05Z

Thank you @cloud-fan @entong ! I will open a new PR to resolve the above comments.

### What changes were proposed in this pull request? Adjust input `format` of function `to_binary`: - gracefully fail for the non-string `format` parameter - remove arguable `base2` format support ### Why are the changes needed? Currently, function to_binary doesn't deal with the non-string `format` parameter properly. For example, `spark.sql("select to_binary('abc', 1)")` raises casting error, rather than hint that encoding format is unsupported. In addition, `base2` format is arguable as discussed [here](#35415 (comment)). We may exclude it following what Snowflake [to_binary](https://docs.snowflake.com/en/sql-reference/functions/to_binary.html) does for now. ### Does this PR introduce _any_ user-facing change? Yes. - Better error messages for non-string `format` parameter. For example: From: ``` scala> spark.sql("select to_binary('abc', 1)") org.apache.spark.sql.AnalysisException: class java.lang.Integer cannot be cast to class org.apache.spark.unsafe.types.UTF8String (java.lang.Integer is in module java.base of loader 'bootstrap'; org.apache.spark.unsafe.types.UTF8String is in unnamed module of loader 'app'); line 1 pos 7 ``` To: ``` scala> spark.sql("select to_binary('abc', 1)") org.apache.spark.sql.AnalysisException: cannot resolve 'to_binary('abc', 1)' due to data type mismatch: Unsupported encoding format: Some(1). The format has to be a case-insensitive string literal of 'hex', 'utf-8', 'base2', or 'base64'; line 1 pos 7; ``` - Removed `base2` format support ``` scala> spark.sql("select to_binary('abc', 'base2')").show() org.apache.spark.sql.AnalysisException: cannot resolve 'to_binary('abc', 'base2')' due to data type mismatch: Unsupported encoding format: Some(base2). The format has to be a case-insensitive string literal of 'hex', 'utf-8', or 'base64'; line 1 pos 7; ``` ### How was this patch tested? Unit test. Closes #35533 from xinrong-databricks/to_binary_followup. Authored-by: Xinrong Meng <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

github-actions bot added the SQL label Feb 7, 2022

xinrong-meng force-pushed the to_binary branch from 720746a to fa925c7 Compare February 9, 2022 04:24

xinrong-meng changed the title ~~[WIP] Add new SQL function to_binary~~ [SPARK-37507][SQL] Add a new SQL function to_binary Feb 9, 2022

xinrong-meng changed the title ~~[SPARK-37507][SQL] Add a new SQL function to_binary~~ [WIP][SPARK-37507][SQL] Add a new SQL function to_binary Feb 9, 2022

HyukjinKwon reviewed Feb 9, 2022

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala Show resolved Hide resolved

HyukjinKwon reviewed Feb 9, 2022

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Feb 9, 2022

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Feb 9, 2022

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala Outdated Show resolved Hide resolved

xinrong-meng force-pushed the to_binary branch 2 times, most recently from a5c5653 to 67c433c Compare February 10, 2022 04:02

HyukjinKwon reviewed Feb 10, 2022

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Feb 10, 2022

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala Outdated Show resolved Hide resolved

xinrong-meng changed the title ~~[WIP][SPARK-37507][SQL] Add a new SQL function to_binary~~ [SPARK-37507][SQL] Add a new SQL function to_binary Feb 10, 2022

xinrong-meng marked this pull request as ready for review February 10, 2022 06:51

xinrong-meng commented Feb 10, 2022

View reviewed changes

HyukjinKwon reviewed Feb 10, 2022

View reviewed changes

sql/core/src/test/resources/sql-tests/results/ansi/string-functions.sql.out Outdated Show resolved Hide resolved

gengliangwang reviewed Feb 10, 2022

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala Show resolved Hide resolved

HyukjinKwon reviewed Feb 11, 2022

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala Outdated Show resolved Hide resolved

xinrong-meng commented Feb 11, 2022

View reviewed changes

gengliangwang approved these changes Feb 11, 2022

View reviewed changes

HyukjinKwon reviewed Feb 11, 2022

View reviewed changes

sql/core/src/test/resources/sql-functions/sql-expression-schema.md Show resolved Hide resolved

xinrong-meng added 8 commits February 11, 2022 20:41

draft

6c47eb0

ExpressionDescription

e4e86c0

checkInputDataTypes

6cf7ad1

format is optional

dbd834a

fix format

71b6bf1

match Literal.toString

7e94782

sql-expr schema

ff5120e

SQLQueryTest input

7887b43

xinrong-meng added 15 commits February 11, 2022 20:41

SQLQueryTest generated out

66f7d49

resolve comments

d6d686c

foldable

e7a5afc

generated out

352ee8f

case-insensitive format

561fde8

generated out

8db575b

format eval null

6036dab

support null input

40d1d01

docstr for NULL

e3a616a

foldable/case-insensitive format test

c22ac3c

better doc

f9e92eb

fix for null format

2d4200c

more tests

3568305

no pipe symbol ExpressionDescription

9180212

fix schema

300f1e8

xinrong-meng force-pushed the to_binary branch from 3b937f1 to 300f1e8 Compare February 11, 2022 12:52

restore format

3fd1b77

HyukjinKwon approved these changes Feb 11, 2022

View reviewed changes

HyukjinKwon closed this in 25dd425 Feb 13, 2022

cloud-fan reviewed Feb 14, 2022

View reviewed changes

entong reviewed Feb 14, 2022

View reviewed changes

cloud-fan reviewed Feb 15, 2022

View reviewed changes

xinrong-meng mentioned this pull request Feb 16, 2022

[SPARK-38225][SQL] Adjust input format of function to_binary #35533

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-37507][SQL] Add a new SQL function to_binary #35415

[SPARK-37507][SQL] Add a new SQL function to_binary #35415

xinrong-meng commented Feb 7, 2022 •

edited

Loading

HyukjinKwon commented Feb 10, 2022

xinrong-meng Feb 10, 2022 •

edited

Loading

xinrong-meng Feb 11, 2022

xinrong-meng commented Feb 11, 2022

HyukjinKwon commented Feb 13, 2022

cloud-fan Feb 14, 2022

xinrong-meng Feb 16, 2022

entong Feb 14, 2022

xinrong-meng Feb 16, 2022 •

edited

Loading

cloud-fan Feb 16, 2022

cloud-fan Feb 15, 2022

xinrong-meng commented Feb 15, 2022

[SPARK-37507][SQL] Add a new SQL function to_binary #35415

[SPARK-37507][SQL] Add a new SQL function to_binary #35415

Conversation

xinrong-meng commented Feb 7, 2022 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

HyukjinKwon commented Feb 10, 2022

xinrong-meng Feb 10, 2022 • edited Loading

Choose a reason for hiding this comment

xinrong-meng Feb 11, 2022

Choose a reason for hiding this comment

xinrong-meng commented Feb 11, 2022

HyukjinKwon commented Feb 13, 2022

cloud-fan Feb 14, 2022

Choose a reason for hiding this comment

xinrong-meng Feb 16, 2022

Choose a reason for hiding this comment

entong Feb 14, 2022

Choose a reason for hiding this comment

xinrong-meng Feb 16, 2022 • edited Loading

Choose a reason for hiding this comment

cloud-fan Feb 16, 2022

Choose a reason for hiding this comment

cloud-fan Feb 15, 2022

Choose a reason for hiding this comment

xinrong-meng commented Feb 15, 2022

xinrong-meng commented Feb 7, 2022 •

edited

Loading

xinrong-meng Feb 10, 2022 •

edited

Loading

xinrong-meng Feb 16, 2022 •

edited

Loading