Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The compatibility issues between Velox and Spark #822

Closed
rui-mo opened this issue Jan 9, 2023 · 2 comments
Closed

The compatibility issues between Velox and Spark #822

rui-mo opened this issue Jan 9, 2023 · 2 comments
Labels
enhancement New feature or request velox backend works for Velox backend

Comments

@rui-mo
Copy link
Contributor

rui-mo commented Jan 9, 2023

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

To solve the compatibility issues between Velox and Spark, there are some changes to Velox. We are tracking them here for future improvement.

Uncompatibility Detailed Information
The handling for dot in file path Disable tokenizing the path by dot: oap-project/velox#109.
Presto sql function substr Fix the result mismatch for right expression: oap-project/velox#107.
Presto sql sum Spark sum can overflow: oap-project/velox#101.
Presto sql avg Align partial avg results: oap-project/velox#56.
Support more datetime pattern Datetime like YY, YY-MM are valid: oap-project/velox#94
Presto sql corr Align corr accuracy with Spark: oap-project/velox#147.
@rui-mo rui-mo added the enhancement New feature or request label Jan 9, 2023
@rui-mo
Copy link
Contributor Author

rui-mo commented Jan 9, 2023

cc @jinchengchenghh

@FelixYBW
Copy link
Contributor

To the functions has different implementation. Let's check if presto has different implementation or not. If presto has different result, we should copy the function to sparksql foder. If presto has the same result, we should modify directly from presto's one. We may check if Velox has a UT for our error case to quickly identify if Presto implements differently

@weiting-chen weiting-chen moved this to 🆕 New in Gluten 0.5.0 Mar 1, 2023
@weiting-chen weiting-chen added the velox backend works for Velox backend label Mar 1, 2023
@rui-mo rui-mo closed this as completed Oct 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request velox backend works for Velox backend
Projects
None yet
Development

No branches or pull requests

3 participants