re matching looks broken for unicode

[kevinjwalters]

CircuitPython version

Adafruit CircuitPython 7.3.3 on 2022-08-29; Adafruit MagTag with ESP32S2

Code/REPL

import re

text1 = "CircuitPython loves regular expressions"
text2 = "CircuitPython \u2764 regular expressions"

regex1 = re.compile("re")

for text in (text1, text2):
    match = regex1.search(text)
    print(text)
    print("Match:", text[match.start():match.end()])
    print()

Behavior

The start() and end() are borked if there's preceeding unicode probably due to flaws in character vs byte counting for variable length unicode/utf world.

Description

No response

Additional information

Output from a MagTag:

Adafruit CircuitPython 7.3.3 on 2022-08-29; Adafruit MagTag with ESP32S2
>>>
soft reboot

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
CircuitPython loves regular expressions
Match: re

CircuitPython ❤ regular expressions
Match: gu


Code done running.

[jepler]

I've reproduced and reported the bug in micropython. Thank you for this report.

[michalpokusa]

I also encountered this issue while working on POC of adafruit_template_engine.

While this is not fixed, one way of making it work is using encode("utf-8") and then decode("utf-8").

text.encode("utf-8")[match.start():match.end()].decode("utf-8")

This is not an elegant solution, but might be the only way of doing it right now.

[jepler]

It's worth noting that we should get micropython's bug fix for this when we merge mp version 1.20, which is planned for circuitpython version 9.

[dhalbert]

Re-tested. This is fixed as of 9.0.0-alpha.2