BUG: 35977 Adding regex support for ExtensionBlock replace method #36038

cgangwar11 · 2020-09-01T14:37:13Z

closes BUG: replacement works for object but not string dtype #35977
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Overrides replace method to add support of regex in ExtensionBlock . If input to_replace is a regex we convert data type of block to Object and calls object replace method which supports regex otherwise we call replace method of super class Block.

In [2]: b.replace(r'^\s*$', pd.NA, regex=True, inplace=True)
   ...: print(b)
   a     b
0  a     d
1  b  <NA>
2  c  <NA>

In [3]: b.replace(r'^\s*$', pd.NA, regex=True, inplace=False)
Out[3]: 
   a     b
0  a     d
1  b  <NA>
2  c  <NA>

…d replacement

…35977

pandas/tests/arrays/categorical/test_replace.py

jreback · 2020-09-02T17:57:53Z

pls merge master as we just merged a change in how replace works in internals.

…35977

cgangwar11 · 2020-09-03T06:12:11Z

@jreback synced with upstream master also verified that this bug is still present in upstream master

…35977

simonjayhawkins · 2020-09-10T14:03:23Z

@cgangwar11

Conflicting files
pandas/tests/arrays/categorical/test_replace.py

jbrockmendel · 2020-10-16T14:46:26Z

pandas/core/internals/blocks.py

+        regex = validate_bool_kwarg(regex, "regex")
+        if regex:
+            dtype = self.values.dtype
+            blocks = self.astype(object)


can you give this a non-plural name

@cgangwar11 can you address?

arw2019 · 2020-10-30T03:46:30Z

doc/source/whatsnew/v1.2.0.rst

 - Fixed Bug where :class:`DataFrame` column set to scalar extension type via a dict instantion was considered an object type rather than the extension type (:issue:`35965`)
 - Fixed bug where ``astype()`` with equal dtype and ``copy=False`` would return a new object (:issue:`284881`)
 - Fixed bug when applying a NumPy ufunc with multiple outputs to a :class:`pandas.arrays.IntegerArray` returning None (:issue:`36913`)
+- Fixed bug in :meth:`Dataframe.replace` not working for ``regex=True`` and ``dtype='string'`` Now working as expected (:issue:`35977`)


Please change this to:

- Fixed bug in :meth:`DataFrame.replace` now returns correct result for `regex=True`` with ``string`` dtype (:issue:`35977`)

arw2019 · 2020-10-30T03:48:18Z

pandas/tests/arrays/categorical/test_replace.py

+)
+def test_replace_string(to_replace, value, input_data, expected_data):
+    # GH35977
+    input_df = pd.DataFrame({"col1": input_data}, dtype="string")


Call this df

arw2019 · 2020-10-30T03:48:50Z

pandas/tests/arrays/categorical/test_replace.py

+    # GH35977
+    input_df = pd.DataFrame({"col1": input_data}, dtype="string")
+    expected_df = pd.DataFrame({"col1": expected_data}, dtype="string")
+    input_df = input_df.replace(to_replace, value, inplace=False, regex=False)


Call this result

arw2019 · 2020-10-30T03:49:04Z

pandas/tests/arrays/categorical/test_replace.py

+def test_replace_string(to_replace, value, input_data, expected_data):
+    # GH35977
+    input_df = pd.DataFrame({"col1": input_data}, dtype="string")
+    expected_df = pd.DataFrame({"col1": expected_data}, dtype="string")


Call this expected

arw2019 · 2020-10-30T03:50:09Z

pandas/tests/arrays/categorical/test_replace.py

+        ("f", "replace", ["d", "ee", "f", ""], ["d", "ee", "replace", ""]),
+    ],
+)
+def test_replace_string(to_replace, value, input_data, expected_data):


Use the idiom:

result = ... expected = ... tm.assert_frame_equal(result, frame)

arw2019 · 2020-10-30T03:51:48Z

pandas/tests/arrays/categorical/test_replace.py

+)
+def test_replace_regex(to_replace, value, input_data, expected_data, inplace):
+    # GH35977
+    input_df = pd.DataFrame({"col1": input_data}, dtype="string")


Call this df

arw2019 · 2020-10-30T03:52:04Z

pandas/tests/arrays/categorical/test_replace.py

+def test_replace_regex(to_replace, value, input_data, expected_data, inplace):
+    # GH35977
+    input_df = pd.DataFrame({"col1": input_data}, dtype="string")
+    expected_df = pd.DataFrame({"col1": expected_data}, dtype="string")


Call this expected

arw2019 · 2020-10-30T03:52:28Z

pandas/tests/arrays/categorical/test_replace.py

+    # GH35977
+    input_df = pd.DataFrame({"col1": input_data}, dtype="string")
+    expected_df = pd.DataFrame({"col1": expected_data}, dtype="string")
+    converted = input_df.replace(to_replace, value, inplace=inplace, regex=True)


Call this df_replaced

arw2019 · 2020-10-30T03:53:28Z

pandas/tests/arrays/categorical/test_replace.py

+    ],
+)
+def test_replace_regex(to_replace, value, input_data, expected_data, inplace):
+    # GH35977


Use the idiom:

result = ... expected = ... tm.assert_frame_equal(result, frame)

arw2019 · 2020-10-30T03:54:14Z

pandas/core/internals/blocks.py

+        regex = validate_bool_kwarg(regex, "regex")
+        if regex:
+            dtype = self.values.dtype
+            blocks = self.astype(object)


@cgangwar11 can you address?

arw2019 · 2020-10-30T03:55:01Z

Also will need to merge master

jbrockmendel · 2020-11-12T03:26:29Z

pandas/tests/arrays/categorical/test_replace.py

+        (r"f", "replace", ["d", "ee", "f", ""], ["d", "ee", "replace", ""], True),
+    ],
+)
+def test_replace_regex(to_replace, value, input_data, expected_data, inplace):


if this is testing DataFrame.replace, these belong in tests.frame.methods.test_replace

pep8speaks · 2020-11-12T07:57:53Z

Hello @cgangwar11! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-11-15 07:14:13 UTC

jbrockmendel · 2020-11-12T19:05:46Z

pandas/tests/frame/methods/test_replace.py

+            ("f", "replace", ["d", "ee", "f", ""], ["d", "ee", "replace", ""]),
+        ],
+    )
+    def test_replace_string(self, to_replace, value, input_data, expected_data):


can you parametrize these using frame_or_series

was this addressed?

jbrockmendel · 2020-11-12T19:07:37Z

pandas/core/internals/blocks.py

+                    for b in block.replace(to_replace, value, inplace, regex)
+                ]
+            block.replace(to_replace, value, inplace, regex)
+            return block.astype(dtype)


We recently implement Block._replace_regex. can you see if we can avoid the object-dtype cast (ideally even this whole method) by using that?

replace_regex is using np.vectorize function internally which gives an error when using not object dtype hence using replace_regex after typecasting.
f = np.vectorize(re_replacer, otypes=[values.dtype])

…35977

…tion

arw2019

@cgangwar11 what's the status of this? it sounds like you addressed most (possibly all) comments so if you merge master and move the whatsnew to 1.3 we'll take another look

One question I have is whether this affects the ExtensionDtype Series.replace case? If yes we'd like to test with series

Another question is whether we have asvs for this? (If not that doesn't have to be part of this PR IMO)

arw2019 · 2020-12-08T18:25:28Z

pandas/tests/frame/methods/test_replace.py

+            ("f", "replace", ["d", "ee", "f", ""], ["d", "ee", "replace", ""]),
+        ],
+    )
+    def test_replace_string(self, to_replace, value, input_data, expected_data):


was this addressed?

github-actions · 2021-01-08T00:30:18Z

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

jreback · 2021-02-11T01:38:59Z

closing as stale. if you want to continue, pls ping and can re-open.

cgangwar11 added 4 commits September 1, 2020 07:39

BUG:35977 add replace method for ExtensionBlock to support regex base…

41c1ee5

…d replacement

Merge branch 'master' of https://github.com/pandas-dev/pandas into GH…

048ed82

…35977

adding whatsnew entry

1fa4935

added validation for input parameters

6dc3604

cgangwar11 changed the title ~~Gh35977~~ Bug: 35977 Adding regex support for ExtensionBlock replace method Sep 1, 2020

cgangwar11 changed the title ~~Bug: 35977 Adding regex support for ExtensionBlock replace method~~ BUG: 35977 Adding regex support for ExtensionBlock replace method Sep 1, 2020

dsaxton reviewed Sep 2, 2020

View reviewed changes

pandas/tests/arrays/categorical/test_replace.py Outdated Show resolved Hide resolved

changes to simplify test cases

de36ab6

jreback added ExtensionArray Extending pandas with custom dtypes or arrays. Internals Related to non-user accessible pandas implementation labels Sep 2, 2020

Merge branch 'master' of https://github.com/pandas-dev/pandas into GH…

690a5a1

…35977

Merge branch 'master' of https://github.com/pandas-dev/pandas into GH…

65fcbb6

…35977

Resolving merge conflicts

8ee0005

simonjayhawkins added the replace replace method label Sep 10, 2020

GYHHAHA mentioned this pull request Sep 19, 2020

BUG: Regex parameter from replace function doesn't work on string dtype #36472

Closed

Merge branch 'master' into GH35977

b19b2af

jbrockmendel reviewed Oct 16, 2020

View reviewed changes

arw2019 suggested changes Oct 30, 2020

View reviewed changes

cgangwar11 added 3 commits November 12, 2020 07:42

Addressed PR comments and synced with master

5fee01e

Resolving PR comments

f565d8b

Resolving Pep8 issues

ec3df78

jbrockmendel reviewed Nov 12, 2020

View reviewed changes

Merge branch 'master' of https://github.com/pandas-dev/pandas into GH…

b56c163

…35977

using block replace_regex method now and dataframe based parameteriza…

93cf495

…tion

arw2019 reviewed Dec 8, 2020

View reviewed changes

github-actions bot added the Stale label Jan 8, 2021

jreback closed this Feb 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: 35977 Adding regex support for ExtensionBlock replace method #36038

BUG: 35977 Adding regex support for ExtensionBlock replace method #36038

cgangwar11 commented Sep 1, 2020

jreback commented Sep 2, 2020

cgangwar11 commented Sep 3, 2020

simonjayhawkins commented Sep 10, 2020

jbrockmendel Oct 16, 2020

arw2019 Oct 30, 2020

arw2019 Oct 30, 2020

arw2019 Oct 30, 2020

arw2019 Oct 30, 2020

arw2019 Oct 30, 2020

arw2019 Oct 30, 2020

arw2019 Oct 30, 2020

arw2019 Oct 30, 2020

arw2019 Oct 30, 2020

arw2019 Oct 30, 2020

arw2019 Oct 30, 2020

arw2019 commented Oct 30, 2020

jbrockmendel Nov 12, 2020

pep8speaks commented Nov 12, 2020 •

edited

Loading

jbrockmendel Nov 12, 2020

arw2019 Dec 8, 2020

jbrockmendel Nov 12, 2020

cgangwar11 Nov 15, 2020

arw2019 left a comment

arw2019 Dec 8, 2020

github-actions bot commented Jan 8, 2021

jreback commented Feb 11, 2021

BUG: 35977 Adding regex support for ExtensionBlock replace method #36038

BUG: 35977 Adding regex support for ExtensionBlock replace method #36038

Conversation

cgangwar11 commented Sep 1, 2020

jreback commented Sep 2, 2020

cgangwar11 commented Sep 3, 2020

simonjayhawkins commented Sep 10, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arw2019 commented Oct 30, 2020

Choose a reason for hiding this comment

pep8speaks commented Nov 12, 2020 • edited Loading

Comment last updated at 2020-11-15 07:14:13 UTC

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arw2019 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 8, 2021

jreback commented Feb 11, 2021

pep8speaks commented Nov 12, 2020 •

edited

Loading