Add array_frequency Presto function #3807

vermapratyush · 2023-01-25T12:19:19Z

Summary:

Test Plan

$ velox_expression_fuzzer_test --  --seed 100 --lazy_vector_generation_ratio 0.2 --duration_sec 600 --enable_variadic_signatures --velox_fuzzer_enable_complex_types --velox_fuzzer_enable_column_reuse --velox_fuzzer_enable_expression_reuse --retry_with_try --logtostderr=1 --minloglevel=0 --repro_persist_path=/tmp/fuzzer_repro --only array_frequency

I0126 12:37:10.402107 2149293 ExpressionFuzzer.cpp:949] ==============================> Done with iteration 22412
I0126 12:37:10.402221 2149293 ExpressionFuzzer.cpp:829] ==============================> Top 1 by number of rows processed
I0126 12:37:10.402263 2149293 ExpressionFuzzer.cpp:831] Format: functionName numTimesSelected proportionOfTimesSelected numProcessedRows
I0126 12:37:10.402277 2149293 ExpressionFuzzer.cpp:835] array_frequency 22413 100.00% 340170
I0126 12:37:10.402318 2149293 ExpressionFuzzer.cpp:841] ==============================> Bottom 1 by number of rows processed
I0126 12:37:10.402338 2149293 ExpressionFuzzer.cpp:843] Format: functionName numTimesSelected proportionOfTimesSelected numProcessedRows
I0126 12:37:10.402352 2149293 ExpressionFuzzer.cpp:848] array_frequency 22413 100.00% 340170
I0126 12:37:10.402372 2149293 ExpressionFuzzer.cpp:861] ==============================> All stats sorted by number of times the function was chosen
I0126 12:37:10.402385 2149293 ExpressionFuzzer.cpp:863] Format: functionName numTimesSelected proportionOfTimesSelected numProcessedRows
I0126 12:37:10.402405 2149293 ExpressionFuzzer.cpp:867] array_frequency 22413 100.00% 340170

Fuzzer on all functions

$ velox_expression_fuzzer_test --  --seed 100 --lazy_vector_generation_ratio 0.2 --duration_sec 600 --enable_variadic_signatures --velox_fuzzer_enable_complex_types --velox_fuzzer_enable_column_reuse --velox_fuzzer_enable_expression_reuse --retry_with_try --logtostderr=1 --minloglevel=0 --repro_persist_path=/tmp/fuzzer_repro

https://pastebin.com/1k4cGMhj

Fixes #3752

Differential Revision: D42712223

netlify · 2023-01-25T12:19:35Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`a38b61f`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/63d418c7e797d500094989d5

facebook-github-bot · 2023-01-25T12:20:00Z

This pull request was exported from Phabricator. Differential Revision: D42712223

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Differential Revision: D42712223 fbshipit-source-id: 6de391726abe4ad7731762874586c2037e94fef1

facebook-github-bot · 2023-01-25T12:26:05Z

This pull request was exported from Phabricator. Differential Revision: D42712223

vermapratyush · 2023-01-25T15:54:11Z

Failures unrelated to the diff.

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Reviewed By: Yuhta Differential Revision: D42712223 fbshipit-source-id: 0a56eb6a0d9ad2fd8cbbac1bdb4aee4d113aba84

facebook-github-bot · 2023-01-25T17:21:06Z

This pull request was exported from Phabricator. Differential Revision: D42712223

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Reviewed By: Yuhta Differential Revision: D42712223 fbshipit-source-id: 4df80d6e6da84de5fdde99450b2110dda74e8f17

facebook-github-bot · 2023-01-25T18:45:43Z

This pull request was exported from Phabricator. Differential Revision: D42712223

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Reviewed By: Yuhta Differential Revision: D42712223 fbshipit-source-id: 2ee3ec58ba717120a83c9d88dbd90eb9e902cc71

facebook-github-bot · 2023-01-25T20:04:50Z

This pull request was exported from Phabricator. Differential Revision: D42712223

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Reviewed By: Yuhta Differential Revision: D42712223 fbshipit-source-id: cf29ac047b6988b5fd72d320887359064ab7573a

facebook-github-bot · 2023-01-26T16:59:54Z

This pull request was exported from Phabricator. Differential Revision: D42712223

netlify · 2023-01-26T16:59:58Z

❌ Deploy Preview for bright-melba-465c6a failed.

Name	Link
🔨 Latest commit	`a38b61f`
🔍 Latest deploy log	https://app.netlify.com/sites/bright-melba-465c6a/deploys/63d418c7f9770d000825450f

mbasmanova

@vermapratyush Thank you for adding one more Presto function. Would you update PR description with results of Fuzzer testing?

Implement array_frequency in velox with same signature as presto.

This statement is redundant. Consider removing.

velox/functions/prestosql/ArrayFunctions.h

mbasmanova · 2023-01-26T16:57:28Z

velox/functions/prestosql/ArrayFunctions.h

+
+    // optimization to skip counting frequency of null values
+    if (inputArray.mayHaveNulls()) {
+      for (const auto& item : inputArray.skipNulls()) {


Any particular reason we can't use this loop in all cases? CC: @laithsakka

@mbasmanova the API is different as .skipNulls also unrolls the value. Hence in the subsequent branch is

frequencyCount[item.value()]++; vs frequencyCount[item]++;

velox/functions/prestosql/tests/ArrayFrequencyTest.cpp

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Reviewed By: Yuhta Differential Revision: D42712223 fbshipit-source-id: aa824bbe6ee32f59344b1c8c08a0c8fdd28e9c6a

facebook-github-bot · 2023-01-26T20:27:35Z

This pull request was exported from Phabricator. Differential Revision: D42712223

vermapratyush · 2023-01-26T20:27:53Z

@mbasmanova Running fuzzer without --only params failed with the following error.
Is this a known issue?

$ buck run //velox/expression/tests:velox_expression_fuzzer_test --  --seed 100 --lazy_vector_generation_ratio 0.2 --duration_sec 600 --enable_variadic_signatures --velox_fuzzer_enable_complex_types --velox_fuzzer_enable_column_reuse --velox_fuzzer_enable_expression_reuse --retry_with_try --logtostderr=1 --minloglevel=0 --repro_persist_path=/tmp/fuzzer_repr

terminate called after throwing an instance of 'facebook::velox::VeloxUserError'
  what():  Exception: VeloxUserError
Error Source: USER
Error Code: UNSUPPORTED
Reason: Unsupported map key type for element_at: INTERVAL DAY TO SECOND
Retriable: False
Function: applyMap
File: buck-out/dev/gen/aab7ed39/velox/functions/lib/velox_functions_lib#header-mode-symlink-tree-with-header-map,headers/velox/functions/lib/SubscriptUtil.h
Line: 144
Stack trace:
Stack trace has been disabled.Use --velox_exception_user_stacktrace=true to enable it.

*** Aborted at 1674764746 (Unix time, try 'date -d @1674764746') ***
*** Signal 6 (SIGABRT) (0x3227d0022efdd) received by PID 2289629 (pthread TID 0x7fc9898acf40) (linux TID 2289629) (maybe from PID 2289629, UID 205437) (code: -6), stack trace: ***
    @ 000000000001100e folly::symbolizer::(anonymous namespace)::innerSignalHandler(int, siginfo_t*, void*)
                       ./folly/experimental/symbolizer/SignalHandler.cpp:449
    @ 000000000000f731 folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       ./folly/experimental/symbolizer/SignalHandler.cpp:470
    @ 000000000004459f (unknown)
    @ 000000000009c9d3 __GI___pthread_kill
    @ 00000000000444ec __GI_raise
    @ 000000000002c432 __GI_abort
    @ 00000000000a3fd4 __gnu_cxx::__verbose_terminate_handler()
    @ 00000000000a1b39 __cxxabiv1::__terminate(void (*)())
    @ 00000000000a1ba4 std::terminate()
    @ 00000000000a1ec1 __cxa_rethrow
    @ 0000000000022874 facebook::velox::test::ExpressionVerifier::verify(std::shared_ptr<facebook::velox::core::ITypedExpr const> const&, std::shared_ptr<facebook::velox::RowVector> const&, std::shared_ptr<facebook::velox::BaseVector>&&, bool, std::vector<unsigned int, std::allocator<unsigned int> >)
                       ./velox/expression/tests/ExpressionVerifier.cpp:190
    @ 000000000009a0f1 facebook::velox::test::ExpressionFuzzer::go()
                       ./velox/expression/tests/ExpressionFuzzer.cpp:941
    @ 00000000000c8ce6 facebook::velox::test::expressionFuzzer(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<facebook::velox::exec::FunctionSignature const*, std::allocator<facebook::velox::exec::FunctionSignature const*> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<facebook::velox::exec::FunctionSignature const*, std::allocator<facebook::velox::exec::FunctionSignature const*> > > > >, unsigned long)
                       ./velox/expression/tests/ExpressionFuzzer.cpp:957
    @ 00000000002e45cf FuzzerRunner::run(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
                       ./velox/expression/tests/FuzzerRunner.h:140
                       -> ./velox/expression/tests/ExpressionFuzzerTest.cpp
    @ 00000000002e3da7 main
                       ./velox/expression/tests/ExpressionFuzzerTest.cpp:67
    @ 000000000002c656 __libc_start_call_main
    @ 000000000002c717 __libc_start_main_alias_2
    @ 00000000002c3580 _start
                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/x86_64/start.S:116

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Reviewed By: Yuhta Differential Revision: D42712223 fbshipit-source-id: b1a37ebe291dc4b6dd9f951705fc00016a02d3c5

facebook-github-bot · 2023-01-26T20:41:43Z

This pull request was exported from Phabricator. Differential Revision: D42712223

mbasmanova · 2023-01-26T21:44:51Z

@vermapratyush This is a known issue that was fixed in #3801

Would you rebase and retry?

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Reviewed By: Yuhta Differential Revision: D42712223 fbshipit-source-id: 36ffcf7c9a337a477ccc21421c8047b49afc9cd8

facebook-github-bot · 2023-01-26T22:39:39Z

This pull request was exported from Phabricator. Differential Revision: D42712223

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Reviewed By: Yuhta Differential Revision: D42712223 fbshipit-source-id: 48fc72c9e09c83c0a246716a9fff56fbf4a990ac

facebook-github-bot · 2023-01-26T23:36:58Z

This pull request was exported from Phabricator. Differential Revision: D42712223

mbasmanova

@vermapratyush Thanks.

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Reviewed By: Yuhta, mbasmanova Differential Revision: D42712223 fbshipit-source-id: 48c73e9ab0574c01945af91f85913628bd5e01ad

facebook-github-bot · 2023-01-26T23:43:29Z

This pull request was exported from Phabricator. Differential Revision: D42712223

facebook-github-bot · 2023-01-27T14:28:48Z

This pull request was exported from Phabricator. Differential Revision: D42712223

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Reviewed By: Yuhta, mbasmanova Differential Revision: D42712223 fbshipit-source-id: 18e54975761581c50b798fef307c50556cdfdfaa

Summary: Pull Request resolved: facebookincubator#3807 Implement `array_frequency` in velox with same signature [as presto](https://prestodb.io/docs/current/functions/array.html#array_frequency). Fixes facebookincubator#3752 Reviewed By: Yuhta, mbasmanova Differential Revision: D42712223 fbshipit-source-id: 8e4f37a040c5c3fff636dbdf473b9d3d38f5e24c

facebook-github-bot · 2023-01-27T18:32:36Z

This pull request was exported from Phabricator. Differential Revision: D42712223

facebook-github-bot · 2023-01-27T19:11:32Z

This pull request has been merged in 9971e7f.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 25, 2023

facebook-github-bot added the fb-exported label Jan 25, 2023

vermapratyush force-pushed the export-D42712223 branch from d90ac0f to 943ae9c Compare January 25, 2023 12:25

vermapratyush force-pushed the export-D42712223 branch from 943ae9c to c8fd497 Compare January 25, 2023 17:21

vermapratyush force-pushed the export-D42712223 branch from c8fd497 to 76eb3a0 Compare January 25, 2023 18:45

vermapratyush force-pushed the export-D42712223 branch from 76eb3a0 to 81fa76a Compare January 25, 2023 20:04

vermapratyush requested a review from mbasmanova January 25, 2023 20:18

vermapratyush force-pushed the export-D42712223 branch from 81fa76a to cf55fb5 Compare January 26, 2023 16:59

mbasmanova reviewed Jan 26, 2023

View reviewed changes

vermapratyush force-pushed the export-D42712223 branch from cf55fb5 to 6e4e024 Compare January 26, 2023 20:27

vermapratyush force-pushed the export-D42712223 branch from 6e4e024 to 2ee28cc Compare January 26, 2023 20:41

vermapratyush requested a review from mbasmanova January 26, 2023 22:36

vermapratyush force-pushed the export-D42712223 branch from 2ee28cc to bb0d6a2 Compare January 26, 2023 22:39

vermapratyush force-pushed the export-D42712223 branch from bb0d6a2 to 02e4a9e Compare January 26, 2023 23:36

mbasmanova approved these changes Jan 26, 2023

View reviewed changes

vermapratyush force-pushed the export-D42712223 branch from 02e4a9e to 14e9932 Compare January 26, 2023 23:43

vermapratyush force-pushed the export-D42712223 branch from 14e9932 to d800e2f Compare January 27, 2023 14:28

vermapratyush force-pushed the export-D42712223 branch from d800e2f to a38b61f Compare January 27, 2023 18:32

facebook-github-bot closed this in 9971e7f Jan 27, 2023

facebook-github-bot added the Merged label Jan 27, 2023

mbasmanova mentioned this pull request Mar 25, 2024

Add Presto functions #2262

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add array_frequency Presto function #3807

Add array_frequency Presto function #3807

vermapratyush commented Jan 25, 2023 •

edited

Loading

netlify bot commented Jan 25, 2023 •

edited

Loading

facebook-github-bot commented Jan 25, 2023

facebook-github-bot commented Jan 25, 2023

vermapratyush commented Jan 25, 2023

facebook-github-bot commented Jan 25, 2023

facebook-github-bot commented Jan 25, 2023

facebook-github-bot commented Jan 25, 2023

facebook-github-bot commented Jan 26, 2023

netlify bot commented Jan 26, 2023 •

edited

Loading

mbasmanova left a comment

mbasmanova Jan 26, 2023

vermapratyush Jan 26, 2023

facebook-github-bot commented Jan 26, 2023

vermapratyush commented Jan 26, 2023

facebook-github-bot commented Jan 26, 2023

mbasmanova commented Jan 26, 2023

facebook-github-bot commented Jan 26, 2023

facebook-github-bot commented Jan 26, 2023

mbasmanova left a comment

facebook-github-bot commented Jan 26, 2023

facebook-github-bot commented Jan 27, 2023

facebook-github-bot commented Jan 27, 2023

facebook-github-bot commented Jan 27, 2023

Add array_frequency Presto function #3807

Add array_frequency Presto function #3807

Conversation

vermapratyush commented Jan 25, 2023 • edited Loading

Test Plan

netlify bot commented Jan 25, 2023 • edited Loading

✅ Deploy Preview for meta-velox canceled.

facebook-github-bot commented Jan 25, 2023

facebook-github-bot commented Jan 25, 2023

vermapratyush commented Jan 25, 2023

facebook-github-bot commented Jan 25, 2023

facebook-github-bot commented Jan 25, 2023

facebook-github-bot commented Jan 25, 2023

facebook-github-bot commented Jan 26, 2023

netlify bot commented Jan 26, 2023 • edited Loading

❌ Deploy Preview for bright-melba-465c6a failed.

mbasmanova left a comment

Choose a reason for hiding this comment

mbasmanova Jan 26, 2023

Choose a reason for hiding this comment

vermapratyush Jan 26, 2023

Choose a reason for hiding this comment

facebook-github-bot commented Jan 26, 2023

vermapratyush commented Jan 26, 2023

facebook-github-bot commented Jan 26, 2023

mbasmanova commented Jan 26, 2023

facebook-github-bot commented Jan 26, 2023

facebook-github-bot commented Jan 26, 2023

mbasmanova left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Jan 26, 2023

facebook-github-bot commented Jan 27, 2023

facebook-github-bot commented Jan 27, 2023

facebook-github-bot commented Jan 27, 2023

vermapratyush commented Jan 25, 2023 •

edited

Loading

netlify bot commented Jan 25, 2023 •

edited

Loading

netlify bot commented Jan 26, 2023 •

edited

Loading