Add rank and dense_rank Spark window function #6289
Closed
+50
−15
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for meta-velox canceled.
|
facebook-github-bot
added
the
CLA Signed
JkSelf
force-pushed
the
rank-window-function
branch
2 times, most recently
from
88aea96 to
cdd649d
Compare
|
@mbasmanova @rui-mo Can you help to review? |
JkSelf
force-pushed
the
rank-window-function
branch
from
cdd649d to
4a16f23
Compare
mbasmanova
reviewed
Aug 28, 2023
There was a problem hiding this comment.
@aditi-pandit @rui-mo Aditi, Rui, would you help review this PR?
rui-mo
reviewed
Aug 28, 2023
JkSelf
force-pushed
the
rank-window-function
branch
from
4a16f23 to
3ccdf4f
Compare
|
@aditi-pandit @rui-mo Do you have any further comment? |
aditi-pandit
approved these changes
Aug 30, 2023
|
Thanks @aditi-pandit. @mbasmanova @rui-mo Do you have any comment? |
mbasmanova
approved these changes
Aug 30, 2023
There was a problem hiding this comment.
@JkSelf Looks good. Please, update PR description to wrap each line at 80 characters.
|
There are CI failures. Please, rebase to the latest to see if these go away. |
JkSelf
force-pushed
the
rank-window-function
branch
2 times, most recently
from
733d8d2 to
ac258a4
Compare
|
@mbasmanova updated PR description. And the CI is passed. Please help to merge. Thanks. |
|
@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
JkSelf
force-pushed
the
rank-window-function
branch
from
ac258a4 to
029f25d
Compare
|
@kevinwilfong merged this pull request in e85f942. |
|
Conbench analyzed the 1 benchmark run on commit There were no benchmark performance regressions. 🎉 The full Conbench report has more details. |
codyschierbeck
pushed a commit
to codyschierbeck/velox
that referenced
this pull request
Sep 27, 2023
Summary: Spark [Rank function](https://github.com/apache/spark/blob/f824d058b14e3c58b1c90f64fefc45fac105c7dd/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L1006) computes the rank of a value in a group of values. The result is one plus the number of rows preceding or equal to the current row in the ordering of the partition. The values will produce gaps in the sequence. Spark [DenseRank function](https://github.com/apache/spark/blob/f824d058b14e3c58b1c90f64fefc45fac105c7dd/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L1044) computes the rank of a value in a group of values. The result is one plus the previously assigned rank value. Unlike Rank function, DenseRank will not produce gaps in the ranking sequence. The difference between sparksql and prestosql is the return type, where the sparksql's return type is integer and the prestosql's is bigint. This PR refer the [nth_value()](https://github.com/facebookincubator/velox/blob/b9be1718a70f3f81d184cd1dc57134552a2ed96a/velox/functions/lib/window/NthValue.h#L20) function and move the Rank.cpp file from velox/functions/prestosql/window into the velox/functions/window. And also provide registerRankBigint and registerRankInteger for prestosql and sparksql. Pull Request resolved: facebookincubator#6289 Reviewed By: laithsakka Differential Revision: D48827183 Pulled By: kevinwilfong fbshipit-source-id: cbf2e5a8da2bef0593304bf630fa32b1fc071559
codyschierbeck
pushed a commit
to codyschierbeck/velox
that referenced
this pull request
Sep 27, 2023
Summary: Spark [Rank function](https://github.com/apache/spark/blob/f824d058b14e3c58b1c90f64fefc45fac105c7dd/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L1006) computes the rank of a value in a group of values. The result is one plus the number of rows preceding or equal to the current row in the ordering of the partition. The values will produce gaps in the sequence. Spark [DenseRank function](https://github.com/apache/spark/blob/f824d058b14e3c58b1c90f64fefc45fac105c7dd/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L1044) computes the rank of a value in a group of values. The result is one plus the previously assigned rank value. Unlike Rank function, DenseRank will not produce gaps in the ranking sequence. The difference between sparksql and prestosql is the return type, where the sparksql's return type is integer and the prestosql's is bigint. This PR refer the [nth_value()](https://github.com/facebookincubator/velox/blob/b9be1718a70f3f81d184cd1dc57134552a2ed96a/velox/functions/lib/window/NthValue.h#L20) function and move the Rank.cpp file from velox/functions/prestosql/window into the velox/functions/window. And also provide registerRankBigint and registerRankInteger for prestosql and sparksql. Pull Request resolved: facebookincubator#6289 Reviewed By: laithsakka Differential Revision: D48827183 Pulled By: kevinwilfong fbshipit-source-id: cbf2e5a8da2bef0593304bf630fa32b1fc071559
codyschierbeck
pushed a commit
to codyschierbeck/velox
that referenced
this pull request
Sep 27, 2023
Summary: Spark [Rank function](https://github.com/apache/spark/blob/f824d058b14e3c58b1c90f64fefc45fac105c7dd/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L1006) computes the rank of a value in a group of values. The result is one plus the number of rows preceding or equal to the current row in the ordering of the partition. The values will produce gaps in the sequence. Spark [DenseRank function](https://github.com/apache/spark/blob/f824d058b14e3c58b1c90f64fefc45fac105c7dd/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L1044) computes the rank of a value in a group of values. The result is one plus the previously assigned rank value. Unlike Rank function, DenseRank will not produce gaps in the ranking sequence. The difference between sparksql and prestosql is the return type, where the sparksql's return type is integer and the prestosql's is bigint. This PR refer the [nth_value()](https://github.com/facebookincubator/velox/blob/b9be1718a70f3f81d184cd1dc57134552a2ed96a/velox/functions/lib/window/NthValue.h#L20) function and move the Rank.cpp file from velox/functions/prestosql/window into the velox/functions/window. And also provide registerRankBigint and registerRankInteger for prestosql and sparksql. Pull Request resolved: facebookincubator#6289 Reviewed By: laithsakka Differential Revision: D48827183 Pulled By: kevinwilfong fbshipit-source-id: cbf2e5a8da2bef0593304bf630fa32b1fc071559
ericyuliu
pushed a commit
to ericyuliu/velox
that referenced
this pull request
Oct 12, 2023
Summary: Spark [Rank function](https://github.com/apache/spark/blob/f824d058b14e3c58b1c90f64fefc45fac105c7dd/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L1006) computes the rank of a value in a group of values. The result is one plus the number of rows preceding or equal to the current row in the ordering of the partition. The values will produce gaps in the sequence. Spark [DenseRank function](https://github.com/apache/spark/blob/f824d058b14e3c58b1c90f64fefc45fac105c7dd/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L1044) computes the rank of a value in a group of values. The result is one plus the previously assigned rank value. Unlike Rank function, DenseRank will not produce gaps in the ranking sequence. The difference between sparksql and prestosql is the return type, where the sparksql's return type is integer and the prestosql's is bigint. This PR refer the [nth_value()](https://github.com/facebookincubator/velox/blob/b9be1718a70f3f81d184cd1dc57134552a2ed96a/velox/functions/lib/window/NthValue.h#L20) function and move the Rank.cpp file from velox/functions/prestosql/window into the velox/functions/window. And also provide registerRankBigint and registerRankInteger for prestosql and sparksql. Pull Request resolved: facebookincubator#6289 Reviewed By: laithsakka Differential Revision: D48827183 Pulled By: kevinwilfong fbshipit-source-id: cbf2e5a8da2bef0593304bf630fa32b1fc071559
99 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Spark Rank function computes the rank of a value in a group of values.
The result is one plus the number of rows preceding or equal to the current row in
the ordering of the partition. The values will produce gaps in the sequence.
Spark DenseRank function computes the rank of a value in a group of values.
The result is one plus the previously assigned rank value. Unlike Rank function,
DenseRank will not produce gaps in the ranking sequence. The difference between sparksql and
prestosql is the return type, where the sparksql's return type is integer and the prestosql's is bigint.
This PR refer the nth_value() function and move the Rank.cpp file from
velox/functions/prestosql/window into the velox/functions/window.
And also provide registerRankBigint and registerRankInteger for prestosql and sparksql.