Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-38225][SQL] Adjust input format of function to_binary #35533

Closed
wants to merge 7 commits into from
Closed

[SPARK-38225][SQL] Adjust input format of function to_binary #35533

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5c01f94
for wrong format input
xinrong-meng Feb 15, 2022
2869cb1
test
xinrong-meng Feb 15, 2022
98634d7
trigger test
xinrong-meng Feb 16, 2022
8be5db9
rmv base2 format support
xinrong-meng Feb 16, 2022
4b93726
fail when format is cast(null as int)
xinrong-meng Feb 17, 2022
6d94e00
rmv unnecessary chk
xinrong-meng Feb 17, 2022
75494c3
fix
xinrong-meng Feb 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions ...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2545,7 +2545,7 @@ case class Encode(value: Expression, charset: Expression)
@ExpressionDescription(
usage = """
_FUNC_(str[, fmt]) - Converts the input `str` to a binary value based on the supplied `fmt`.
`fmt` can be a case-insensitive string literal of "hex", "utf-8", "base2", or "base64".
`fmt` can be a case-insensitive string literal of "hex", "utf-8", or "base64".
By default, the binary format for conversion is "hex" if `fmt` is omitted.
The function returns NULL if at least one of the input parameters is NULL.
""",
Expand All @@ -2562,18 +2562,17 @@ case class ToBinary(expr: Expression, format: Option[Expression], child: Express

def this(expr: Expression, format: Expression) = this(expr, Option(format),
format match {
case lit if lit.foldable =>
case lit if (lit.foldable && Seq(StringType, NullType).contains(lit.dataType)) =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, do other systems like snowflake support to_binary('abc', null)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake doesn't specify that in their document, but BigQuery and Teradata support to_binary('abc', null):

Expressions, functions, and operators | BigQuery | Google Cloud "If an operand is NULL, the result is NULL, with the exception of the IS operator."

Teradata Online Documentation | Quick access to technical manuals "If either instring or in_encoding is NULL, the result is NULL."

val value = lit.eval()
if (value == null) Literal(null, BinaryType)
else {
else if (value.isInstanceOf[UTF8String]) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible for value to be an instance of java.lang.String?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, it can only be UTF8String

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems we can remove this check now, as we have checked the data type ahead

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense! Removed.

value.asInstanceOf[UTF8String].toString.toLowerCase(Locale.ROOT) match {
case "hex" => Unhex(expr)
case "utf-8" => Encode(expr, Literal("UTF-8"))
case "base64" => UnBase64(expr)
case "base2" => Cast(expr, BinaryType)
case _ => lit
}
}
} else lit

case other => other
}
Expand All @@ -2589,10 +2588,11 @@ case class ToBinary(expr: Expression, format: Option[Expression], child: Express

override def checkInputDataTypes(): TypeCheckResult = {
def checkFormat(lit: Expression) = {
if (lit.foldable) {
if (lit.foldable && Seq(StringType, NullType).contains(lit.dataType)) {
val value = lit.eval()
value == null || Seq("hex", "utf-8", "base64", "base2").contains(
value.asInstanceOf[UTF8String].toString.toLowerCase(Locale.ROOT))
value == null || (value.isInstanceOf[UTF8String] &&
Seq("hex", "utf-8", "base64").contains(
value.asInstanceOf[UTF8String].toString.toLowerCase(Locale.ROOT)))
} else false
}

Expand All @@ -2601,7 +2601,7 @@ case class ToBinary(expr: Expression, format: Option[Expression], child: Express
} else {
TypeCheckResult.TypeCheckFailure(
s"Unsupported encoding format: $format. The format has to be " +
s"a case-insensitive string literal of 'hex', 'utf-8', 'base2', or 'base64'")
s"a case-insensitive string literal of 'hex', 'utf-8', or 'base64'")
}
}

Expand Down
3 changes: 2 additions & 1 deletion sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,6 @@ select to_number('00,454.8-', '00,000.9-');
select to_binary('abc');
select to_binary('abc', 'utf-8');
select to_binary('abc', 'base64');
select to_binary('abc', 'base2');
select to_binary('abc', 'hex');
select to_binary('abc', concat('utf', '-8'));
select to_binary('abc', concat('base', '64'));
Expand All @@ -150,4 +149,6 @@ select to_binary('abc', null);
select to_binary(null, 'utf-8');
select to_binary(null, null);
select to_binary(null, cast(null as string));
select to_binary(null, cast(null as int));
select to_binary('abc', 'invalidFormat');
select to_binary('abc', 1);
30 changes: 20 additions & 10 deletions sql/core/src/test/resources/sql-tests/results/ansi/string-functions.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 116
-- Number of queries: 117


-- !query
Expand Down Expand Up @@ -850,14 +850,6 @@ struct<to_binary(abc, base64):binary>
i�


-- !query
select to_binary('abc', 'base2')
-- !query schema
struct<to_binary(abc, base2):binary>
-- !query output
abc


-- !query
select to_binary('abc', 'hex')
-- !query schema
Expand Down Expand Up @@ -930,10 +922,28 @@ struct<to_binary(NULL, CAST(NULL AS STRING)):binary>
NULL


-- !query
select to_binary(null, cast(null as int))
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.AnalysisException
cannot resolve 'to_binary(NULL, CAST(NULL AS INT))' due to data type mismatch: Unsupported encoding format: Some(ansi_cast(null as int)). The format has to be a case-insensitive string literal of 'hex', 'utf-8', or 'base64'; line 1 pos 7


-- !query
select to_binary('abc', 'invalidFormat')
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.AnalysisException
cannot resolve 'to_binary('abc', 'invalidFormat')' due to data type mismatch: Unsupported encoding format: Some(invalidFormat). The format has to be a case-insensitive string literal of 'hex', 'utf-8', 'base2', or 'base64'; line 1 pos 7
cannot resolve 'to_binary('abc', 'invalidFormat')' due to data type mismatch: Unsupported encoding format: Some(invalidFormat). The format has to be a case-insensitive string literal of 'hex', 'utf-8', or 'base64'; line 1 pos 7


-- !query
select to_binary('abc', 1)
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.AnalysisException
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentionally failed query.

cannot resolve 'to_binary('abc', 1)' due to data type mismatch: Unsupported encoding format: Some(1). The format has to be a case-insensitive string literal of 'hex', 'utf-8', or 'base64'; line 1 pos 7
30 changes: 20 additions & 10 deletions sql/core/src/test/resources/sql-tests/results/string-functions.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 116
-- Number of queries: 117


-- !query
Expand Down Expand Up @@ -846,14 +846,6 @@ struct<to_binary(abc, base64):binary>
i�


-- !query
select to_binary('abc', 'base2')
-- !query schema
struct<to_binary(abc, base2):binary>
-- !query output
abc


-- !query
select to_binary('abc', 'hex')
-- !query schema
Expand Down Expand Up @@ -926,10 +918,28 @@ struct<to_binary(NULL, CAST(NULL AS STRING)):binary>
NULL


-- !query
select to_binary(null, cast(null as int))
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.AnalysisException
cannot resolve 'to_binary(NULL, CAST(NULL AS INT))' due to data type mismatch: Unsupported encoding format: Some(cast(null as int)). The format has to be a case-insensitive string literal of 'hex', 'utf-8', or 'base64'; line 1 pos 7


-- !query
select to_binary('abc', 'invalidFormat')
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.AnalysisException
cannot resolve 'to_binary('abc', 'invalidFormat')' due to data type mismatch: Unsupported encoding format: Some(invalidFormat). The format has to be a case-insensitive string literal of 'hex', 'utf-8', 'base2', or 'base64'; line 1 pos 7
cannot resolve 'to_binary('abc', 'invalidFormat')' due to data type mismatch: Unsupported encoding format: Some(invalidFormat). The format has to be a case-insensitive string literal of 'hex', 'utf-8', or 'base64'; line 1 pos 7


-- !query
select to_binary('abc', 1)
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.AnalysisException
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentionally failed query.

cannot resolve 'to_binary('abc', 1)' due to data type mismatch: Unsupported encoding format: Some(1). The format has to be a case-insensitive string literal of 'hex', 'utf-8', or 'base64'; line 1 pos 7