[SPARK-24884][SQL] add regexp_extract_all support #21985

xueyumusic · 2018-08-03T13:38:13Z

What changes were proposed in this pull request?

This PR tried to add regexp_extract_all sql function. It is supported in presto and pig
Also it refactored regexp_extract since there were some duplicated codes.

Why are the changes needed?

Added regexp_extract_all function

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test

xuanyuanking · 2018-08-04T15:58:56Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala

+          $groupArray.add(UTF8String.fromString($matchResult.group($idx)));
+        }
+      }
+      ${ev.value} = new $arrayClass($groupArray.toArray(new UTF8String[$groupArray.size()]));


Do we need consider about setting ev.isNull?

xuanyuanking · 2018-08-04T16:00:43Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala

+      > SELECT _FUNC_('100-200,300-400', '(\\d+)-(\\d+)', 1);
+       [100, 300]
+  """)
+case class RegExpExtractAll(subject: Expression, regexp: Expression, idx: Expression)


Add an abstract class to reduce duplicated code between RegExpExtractAll and RegExpExtract?

kiszk · 2019-07-19T18:39:53Z

@xueyumusic gentle ping

xueyumusic · 2019-07-21T03:39:50Z

Thanks for ping, @kiszk , I will update the code these days.

afsanjar · 2019-08-24T22:32:31Z

thanks for the patch @xueyumusic, we have been waiting for such feature in Spark for awhile :) any update when your hard work will be publicly available.

xueyumusic · 2019-08-25T14:03:58Z

Hi, @kiszk @afsanjar very sorry for long delay update. I updated to address feedback of @xuanyuanking , thanks

kiszk · 2019-08-25T14:50:52Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala

+    }
+
+    (classNamePattern, matcher, matchResult, termLastRegex, termPattern, setEvNotNull)
+


nit: delete empty line

kiszk · 2019-08-25T14:56:29Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala

+
+    nullSafeCodeGen(ctx, ev, (subject, regexp, idx) => {
+      s"""
+      if (!$regexp.equals($termLastRegex)) {


nit: could you please use the following pattern?

s""" | | """

Thanks, @kiszk , updated.

maropu · 2019-08-26T07:11:23Z

We recently updated the PR template: https://github.com/apache/spark/blob/master/.github/PULL_REQUEST_TEMPLATE, so could you please follow that, first?

maropu · 2019-08-26T07:11:40Z

ok to test

maropu · 2019-08-26T07:13:39Z

Also, could you check if the other databases support the similar function, e.g., oracle, potgresql, mysql, ..., and then describe the result in the PR description?

SparkQA · 2019-08-26T10:52:12Z

Test build #109723 has finished for PR 21985 at commit 4373a82.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class RegExpExtract(subject: Expression, regexp: Expression, idx: Expression)
case class RegExpExtractAll(subject: Expression, regexp: Expression, idx: Expression)

xueyumusic · 2019-08-26T13:10:22Z

Hi, @maropu , I updated PR description based on new template, thanks.

kiszk · 2019-09-03T18:26:23Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala

+  protected def getMatcher(s: Any, p: Any) = {
+    if (!p.equals(lastRegex)) {
+      // regex value changed
+      lastRegex = p.asInstanceOf[UTF8String].clone()


~~nit: Do we need to clone p?~~
nvm, it follows the original code

maropu · 2019-09-04T01:52:00Z

Also, could you check if the other databases support the similar function, e.g., oracle, potgresql, mysql, ..., and then describe the result in the PR description?

Have you checked this? pig and presto only support this function?

AmplabJenkins · 2019-09-16T18:20:19Z

Can one of the admins verify this patch?

HyukjinKwon · 2019-09-17T00:13:41Z

ping @xueyumusic

justgfather · 2019-09-26T20:18:12Z

Hey, was this feature merged in the master. If yes, in what version of spark?

kiszk · 2019-09-27T07:05:40Z

@justgfather Not merged yet. If this would be merged now, you can use this function in Spark 3.0.

github-actions · 2020-01-08T00:07:36Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

gatorsmile · 2020-02-08T05:00:06Z

cc @beliefer Could you take this over?

beliefer · 2020-02-08T05:15:42Z

@gatorsmile OK.

### What changes were proposed in this pull request? `regexp_extract_all` is a very useful function expanded the capabilities of `regexp_extract`. There are some description of this function. ``` SELECT regexp_extract('1a 2b 14m', '\d+', 0); -- 1 SELECT regexp_extract_all('1a 2b 14m', '\d+', 0); -- [1, 2, 14] SELECT regexp_extract('1a 2b 14m', '(\d+)([a-z]+)', 2); -- 'a' SELECT regexp_extract_all('1a 2b 14m', '(\d+)([a-z]+)', 2); -- ['a', 'b', 'm'] ``` There are some mainstream database support the syntax. **Presto:** https://prestodb.io/docs/current/functions/regexp.html **Pig:** https://pig.apache.org/docs/latest/api/org/apache/pig/builtin/REGEX_EXTRACT_ALL.html Note: This PR pick up the work of #21985 ### Why are the changes needed? `regexp_extract_all` is a very useful function and make work easier. ### Does this PR introduce any user-facing change? No ### How was this patch tested? New UT Closes #27507 from beliefer/support-regexp_extract_all. Lead-authored-by: beliefer <[email protected]> Co-authored-by: gengjiaan <[email protected]> Co-authored-by: Jiaan Geng <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

xuanyuanking reviewed Aug 4, 2018

View reviewed changes

dongjoon-hyun added the SQL label Jun 14, 2019

RegExpExtractAll

8a72c6c

feedback address

2707d59

xueyumusic force-pushed the RegExpExtractAll branch from 2a96238 to 2707d59 Compare August 25, 2019 13:51

Merge branch 'master' into RegExpExtractAll

a1c7e95

kiszk reviewed Aug 25, 2019

View reviewed changes

feedback address

4373a82

kiszk reviewed Sep 3, 2019

View reviewed changes

github-actions bot added the Stale label Jan 8, 2020

maropu closed this Jan 8, 2020

beliefer mentioned this pull request Feb 9, 2020

[SPARK-24884][SQL] Support regexp function regexp_extract_all #27507

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-24884][SQL] add regexp_extract_all support #21985

[SPARK-24884][SQL] add regexp_extract_all support #21985

xueyumusic commented Aug 3, 2018 •

edited

Loading

xuanyuanking Aug 4, 2018

xuanyuanking Aug 4, 2018

kiszk commented Jul 19, 2019

xueyumusic commented Jul 21, 2019

afsanjar commented Aug 24, 2019

xueyumusic commented Aug 25, 2019

kiszk Aug 25, 2019

kiszk Aug 25, 2019

xueyumusic Aug 26, 2019

maropu commented Aug 26, 2019

maropu commented Aug 26, 2019

maropu commented Aug 26, 2019

SparkQA commented Aug 26, 2019

xueyumusic commented Aug 26, 2019

kiszk Sep 3, 2019 •

edited

Loading

maropu commented Sep 4, 2019 •

edited

Loading

AmplabJenkins commented Sep 16, 2019

HyukjinKwon commented Sep 17, 2019

justgfather commented Sep 26, 2019

kiszk commented Sep 27, 2019

github-actions bot commented Jan 8, 2020

gatorsmile commented Feb 8, 2020

beliefer commented Feb 8, 2020

		}

		(classNamePattern, matcher, matchResult, termLastRegex, termPattern, setEvNotNull)

[SPARK-24884][SQL] add regexp_extract_all support #21985

[SPARK-24884][SQL] add regexp_extract_all support #21985

Conversation

xueyumusic commented Aug 3, 2018 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

xuanyuanking Aug 4, 2018

Choose a reason for hiding this comment

xuanyuanking Aug 4, 2018

Choose a reason for hiding this comment

kiszk commented Jul 19, 2019

xueyumusic commented Jul 21, 2019

afsanjar commented Aug 24, 2019

xueyumusic commented Aug 25, 2019

kiszk Aug 25, 2019

Choose a reason for hiding this comment

kiszk Aug 25, 2019

Choose a reason for hiding this comment

xueyumusic Aug 26, 2019

Choose a reason for hiding this comment

maropu commented Aug 26, 2019

maropu commented Aug 26, 2019

maropu commented Aug 26, 2019

SparkQA commented Aug 26, 2019

xueyumusic commented Aug 26, 2019

kiszk Sep 3, 2019 • edited Loading

Choose a reason for hiding this comment

maropu commented Sep 4, 2019 • edited Loading

AmplabJenkins commented Sep 16, 2019

HyukjinKwon commented Sep 17, 2019

justgfather commented Sep 26, 2019

kiszk commented Sep 27, 2019

github-actions bot commented Jan 8, 2020

gatorsmile commented Feb 8, 2020

beliefer commented Feb 8, 2020

xueyumusic commented Aug 3, 2018 •

edited

Loading

kiszk Sep 3, 2019 •

edited

Loading

maropu commented Sep 4, 2019 •

edited

Loading