Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the result mismatch for right expression #107

Merged
merged 1 commit into from
Jan 4, 2023

Conversation

JkSelf
Copy link
Collaborator

@JkSelf JkSelf commented Jan 4, 2023

For right("ab", 3) expression, vanilla spark will convert the expression to substring("ab", -3). When offload to velox, the start index will be negative. And then the result will be empty, which is not same with vanilla spark "ab". This PR is aim to keep the result same with vanilla spark.

@JkSelf JkSelf requested a review from rui-mo January 4, 2023 08:49
if (start <= 0) {
start = 1;
length = numCharacters;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a final solution is to implement this function under sparksql folder.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Presto has such UT? If presto use the same logic, then we can modify from here directly. If Presto doesn't have the check, we should copy the function to sparksql folder

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presto doesn't have this check. And I will put this function into sparksql folder later.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we may submit an issue to Velox firstly and ask if presto should have this check or not before we put it in sparksql function.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presto use the following check and will return empty result, which is different with vanilla spark.

   // Following Presto semantics
   if (start <= 0 || start > numCharacters || length <= 0) {
     result.setEmpty();
     return true;
   }

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. In this way we must re-implement the function in sparksql

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I will file a new PR to add this function in sparksql.

@rui-mo rui-mo merged commit 6364b1e into oap-project:main Jan 4, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Jan 5, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Feb 27, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 6, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 27, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 29, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 29, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 14, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 17, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 19, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants