Move VectorReader test to its own file, fixed some typos (#1330) #25
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D30024069 |
christycylee
force-pushed
the
export-D30024069
branch
from
ef46b46 to
8649958
Compare
|
This pull request was exported from Phabricator. Differential Revision: D30024069 |
Summary: velox/experimental/codegen did not have its CMakeLists.txt set up properly, so some tests weren't ran when building with cmake. I had to disable one test, that will be addressed in the future. Pull Request resolved: facebookincubator/velox#26 Differential Revision: D30262438 Pulled By: christycylee fbshipit-source-id: a47f4f08ccc91fc85d72d1a3924c1540edc012b6
Summary: Pull Request resolved: facebookincubator/velox#25 Since we'll be adding multiple VectorReaders, I think it's better organization to put the tests in its own directory. Reviewed By: kkeita Differential Revision: D30024069 fbshipit-source-id: 095e665430d4aad50c2ca819a9d58cc15f74407d
|
This pull request was exported from Phabricator. Differential Revision: D30024069 |
christycylee
force-pushed
the
export-D30024069
branch
from
8649958 to
a77b4ef
Compare
|
This pull request has been merged in a693315. |
rui-mo
pushed a commit
to rui-mo/velox
that referenced
this pull request
Jun 29, 2022
A quick workaround for year registration
rui-mo
added a commit
to rui-mo/velox
that referenced
this pull request
Mar 17, 2023
facebook-github-bot
referenced
this pull request
Jul 2, 2024
…ugh auto-vectorization (#10301) Summary: Improve the performance of hash Spark functions for scalar types through auto-vectorization. We can observe a measurable speed-up with this change: TINYINT: ~4X, SMALLINT: ~4X, BIGINT: ~2X, REAL: ~4X, DOUBLE:~2X, DECIMAL(18, 6): ~2X, DECIMAL(38, 6): ~1.2X, TIMESTAMP: ~1.8X, VARCHAR: ~1.3X. Benchmark result before this optimization: ``` ============================================================================ [...]hmarks/ExpressionBenchmarkBuilder.cpp relative time/iter iters/s ============================================================================ hash#ARRAY<MAP<INTEGER,VARCHAR>>#0%nulls##hash 8.33ms 120.03 hash#ARRAY<MAP<INTEGER,VARCHAR>>#0%nulls##xxhas 8.62ms 116.06 ---------------------------------------------------------------------------- hash#ARRAY<MAP<INTEGER,VARCHAR>>https://github.com/facebookincubator/velox/issues/25%nulls##hash 4.76ms 210.08 hash#ARRAY<MAP<INTEGER,VARCHAR>>https://github.com/facebookincubator/velox/issues/25%nulls##xxha 5.17ms 193.50 ---------------------------------------------------------------------------- hash#BIGINT#0%nulls##hash 1.42ms 706.16 hash#BIGINT#0%nulls##xxhash64 1.44ms 695.32 ---------------------------------------------------------------------------- hash#BIGINT#25%nulls##hash 1.26ms 791.56 hash#BIGINT#25%nulls##xxhash64 1.28ms 779.92 ---------------------------------------------------------------------------- hash#DECIMAL(18, 6)#0%nulls##hash 1.41ms 708.28 hash#DECIMAL(18, 6)#0%nulls##xxhash64 1.44ms 694.50 ---------------------------------------------------------------------------- hash#DECIMAL(18, 6)https://github.com/facebookincubator/velox/issues/25%nulls##hash 1.26ms 792.73 hash#DECIMAL(18, 6)https://github.com/facebookincubator/velox/issues/25%nulls##xxhash64 1.28ms 783.60 ---------------------------------------------------------------------------- hash#DECIMAL(38, 6)#0%nulls##hash 3.26ms 306.97 hash#DECIMAL(38, 6)#0%nulls##xxhash64 3.10ms 322.95 ---------------------------------------------------------------------------- hash#DECIMAL(38, 6)https://github.com/facebookincubator/velox/issues/25%nulls##hash 2.66ms 376.15 hash#DECIMAL(38, 6)https://github.com/facebookincubator/velox/issues/25%nulls##xxhash64 2.52ms 396.68 ---------------------------------------------------------------------------- hash#DOUBLE#0%nulls##hash 1.49ms 670.95 hash#DOUBLE#0%nulls##xxhash64 1.55ms 646.38 ---------------------------------------------------------------------------- hash#DOUBLE#25%nulls##hash 1.33ms 750.15 hash#DOUBLE#25%nulls##xxhash64 1.36ms 737.38 ---------------------------------------------------------------------------- hash#REAL#0%nulls##hash 1.40ms 714.23 hash#REAL#0%nulls##xxhash64 1.53ms 654.87 ---------------------------------------------------------------------------- hash#REAL#25%nulls##hash 1.24ms 804.34 hash#REAL#25%nulls##xxhash64 1.33ms 750.28 ---------------------------------------------------------------------------- hash#ROW<f_map:MAP<INTEGER,VARCHAR>,f_array:ARR 15.46ms 64.68 hash#ROW<f_map:MAP<INTEGER,VARCHAR>,f_array:ARR 14.41ms 69.42 ---------------------------------------------------------------------------- hash#ROW<f_map:MAP<INTEGER,VARCHAR>,f_array:ARR 9.19ms 108.82 hash#ROW<f_map:MAP<INTEGER,VARCHAR>,f_array:ARR 8.97ms 111.52 ---------------------------------------------------------------------------- hash#SMALLINT#0%nulls##hash 1.30ms 769.77 hash#SMALLINT#0%nulls##xxhash64 1.38ms 724.97 ---------------------------------------------------------------------------- hash#SMALLINT#25%nulls##hash 1.16ms 865.78 hash#SMALLINT#25%nulls##xxhash64 1.22ms 820.72 ---------------------------------------------------------------------------- hash#TIMESTAMP#0%nulls##hash 2.04ms 490.14 hash#TIMESTAMP#0%nulls##xxhash64 2.02ms 495.06 ---------------------------------------------------------------------------- hash#TIMESTAMP#25%nulls##hash 1.70ms 587.79 hash#TIMESTAMP#25%nulls##xxhash64 1.71ms 584.17 ---------------------------------------------------------------------------- hash#TINYINT#0%nulls##hash 1.30ms 768.66 hash#TINYINT#0%nulls##xxhash64 1.41ms 707.97 ---------------------------------------------------------------------------- hash#TINYINT#25%nulls##hash 1.16ms 865.09 hash#TINYINT#25%nulls##xxhash64 1.21ms 823.21 ---------------------------------------------------------------------------- hash#VARCHAR#0%nulls##hash 4.65ms 215.03 hash#VARCHAR#0%nulls##xxhash64 4.06ms 246.43 ---------------------------------------------------------------------------- hash#VARCHAR#25%nulls##hash 3.79ms 263.59 hash#VARCHAR#25%nulls##xxhash64 3.35ms 298.64 ---------------------------------------------------------------------------- ``` Benchmark result before after optimization: ``` ============================================================================ [...]hmarks/ExpressionBenchmarkBuilder.cpp relative time/iter iters/s ============================================================================ hash#ARRAY<MAP<INTEGER,VARCHAR>>#0%nulls##hash 7.97ms 125.46 hash#ARRAY<MAP<INTEGER,VARCHAR>>#0%nulls##xxhas 8.09ms 123.62 ---------------------------------------------------------------------------- hash#ARRAY<MAP<INTEGER,VARCHAR>>https://github.com/facebookincubator/velox/issues/25%nulls##hash 4.15ms 240.78 hash#ARRAY<MAP<INTEGER,VARCHAR>>https://github.com/facebookincubator/velox/issues/25%nulls##xxha 4.52ms 221.41 ---------------------------------------------------------------------------- hash#BIGINT#0%nulls##hash 632.90us 1.58K hash#BIGINT#0%nulls##xxhash64 550.34us 1.82K ---------------------------------------------------------------------------- hash#BIGINT#25%nulls##hash 631.35us 1.58K hash#BIGINT#25%nulls##xxhash64 528.26us 1.89K ---------------------------------------------------------------------------- hash#DECIMAL(18, 6)#0%nulls##hash 620.12us 1.61K hash#DECIMAL(18, 6)#0%nulls##xxhash64 542.33us 1.84K ---------------------------------------------------------------------------- hash#DECIMAL(18, 6)https://github.com/facebookincubator/velox/issues/25%nulls##hash 646.62us 1.55K hash#DECIMAL(18, 6)https://github.com/facebookincubator/velox/issues/25%nulls##xxhash64 531.15us 1.88K ---------------------------------------------------------------------------- hash#DECIMAL(38, 6)#0%nulls##hash 2.21ms 452.26 hash#DECIMAL(38, 6)#0%nulls##xxhash64 2.24ms 447.31 ---------------------------------------------------------------------------- hash#DECIMAL(38, 6)https://github.com/facebookincubator/velox/issues/25%nulls##hash 1.98ms 504.33 hash#DECIMAL(38, 6)https://github.com/facebookincubator/velox/issues/25%nulls##xxhash64 1.79ms 559.13 ---------------------------------------------------------------------------- hash#DOUBLE#0%nulls##hash 697.29us 1.43K hash#DOUBLE#0%nulls##xxhash64 555.24us 1.80K ---------------------------------------------------------------------------- hash#DOUBLE#25%nulls##hash 715.82us 1.40K hash#DOUBLE#25%nulls##xxhash64 573.07us 1.74K ---------------------------------------------------------------------------- hash#REAL#0%nulls##hash 241.69us 4.14K hash#REAL#0%nulls##xxhash64 461.39us 2.17K ---------------------------------------------------------------------------- hash#REAL#25%nulls##hash 548.04us 1.82K hash#REAL#25%nulls##xxhash64 561.36us 1.78K ---------------------------------------------------------------------------- hash#ROW<f_map:MAP<INTEGER,VARCHAR>,f_array:ARR 13.80ms 72.46 hash#ROW<f_map:MAP<INTEGER,VARCHAR>,f_array:ARR 13.90ms 71.97 ---------------------------------------------------------------------------- hash#ROW<f_map:MAP<INTEGER,VARCHAR>,f_array:ARR 8.69ms 115.07 hash#ROW<f_map:MAP<INTEGER,VARCHAR>,f_array:ARR 8.49ms 117.74 ---------------------------------------------------------------------------- hash#SMALLINT#0%nulls##hash 232.45us 4.30K hash#SMALLINT#0%nulls##xxhash64 429.04us 2.33K ---------------------------------------------------------------------------- hash#SMALLINT#25%nulls##hash 468.51us 2.13K hash#SMALLINT#25%nulls##xxhash64 482.70us 2.07K ---------------------------------------------------------------------------- hash#TIMESTAMP#0%nulls##hash 1.07ms 936.71 hash#TIMESTAMP#0%nulls##xxhash64 979.66us 1.02K ---------------------------------------------------------------------------- hash#TIMESTAMP#25%nulls##hash 999.56us 1.00K hash#TIMESTAMP#25%nulls##xxhash64 873.88us 1.14K ---------------------------------------------------------------------------- hash#TINYINT#0%nulls##hash 235.68us 4.24K hash#TINYINT#0%nulls##xxhash64 420.17us 2.38K ---------------------------------------------------------------------------- hash#TINYINT#25%nulls##hash 468.88us 2.13K hash#TINYINT#25%nulls##xxhash64 469.59us 2.13K ---------------------------------------------------------------------------- hash#VARCHAR#0%nulls##hash 3.40ms 293.97 hash#VARCHAR#0%nulls##xxhash64 2.97ms 336.20 ---------------------------------------------------------------------------- hash#VARCHAR#25%nulls##hash 2.89ms 346.43 hash#VARCHAR#25%nulls##xxhash64 2.86ms 349.69 ---------------------------------------------------------------------------- ``` Pull Request resolved: #10301 Reviewed By: Yuhta Differential Revision: D59252777 Pulled By: bikramSingh91 fbshipit-source-id: 5ad516fe0df82df6fc0c1568bb04a0fd63664d92
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Pull Request resolved: https://github.com/facebookexternal/f4d/pull/1330
Since we'll be adding multiple VectorReaders, I think it's better organization to put the tests in its own directory.
Differential Revision: D30024069