-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix the result mismatch for right expression #107
Conversation
if (start <= 0) { | ||
start = 1; | ||
length = numCharacters; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe a final solution is to implement this function under sparksql folder.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does Presto has such UT? If presto use the same logic, then we can modify from here directly. If Presto doesn't have the check, we should copy the function to sparksql folder
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Presto doesn't have this check. And I will put this function into sparksql folder later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we may submit an issue to Velox firstly and ask if presto should have this check or not before we put it in sparksql function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Presto use the following check and will return empty result, which is different with vanilla spark.
// Following Presto semantics
if (start <= 0 || start > numCharacters || length <= 0) {
result.setEmpty();
return true;
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. In this way we must re-implement the function in sparksql
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. I will file a new PR to add this function in sparksql.
For
right("ab", 3)
expression, vanilla spark will convert the expression to substring("ab", -3). When offload to velox, the start index will be negative. And then the result will be empty, which is not same with vanilla spark "ab". This PR is aim to keep the result same with vanilla spark.