Autofix for I001 unexpectedly altering characters from Unicode Block “Letterlike Symbols”

[namurphy]

With ruff 0.3.4, I ran into unexpected behavior where the autofix for ruff rule I001 is now altering some characters from Unicode Block “Letterlike Symbols” (U+2100). I suspect that this is related to #10412. 🤔 This might not be the only Unicode block that is affected by this.

For example, ℏ (U+210F; which represents Planck's constant over $2π$) is changed to ħ (U+0127; Latin Small Letter H with Stroke). To reproduce this, I created a file called hbar.py that contains:

from astropy.constants import hbar as ℏ
from numpy import pi as π

h = 2 * π * ℏ

After I ran:

ruff check hbar.py  --select=I001 --fix

I did a git diff and got this:

@@ -1,4 +1,4 @@
-from astropy.constants import hbar as ℏ
+from astropy.constants import hbar as ħ
 from numpy import pi as π

Similarly, if I apply I001 to a file containing a bunch of characters from that block:

import numpy as ℂℇℊℋℌℍℎℐℑℒℓℕℤΩℨKÅℬℭℯℰℱℹℴ

then the diff is

@@ -1 +1 @@
-import numpy as ℂℇℊℋℌℍℎℐℑℒℓℕℤΩℨKÅℬℭℯℰℱℹℴ
+import numpy as CƐgHHHhIILlNZΩZKÅBCeEFio

My expectation was for ruff to not change variable names that are valid Python names, except for rules that are designed specifically to make these changes (e.g., RUF001, RUF002, RUF003).

Thank you again for creating a wonderful tool!

[zanieb]

Thanks for the clear write-up!

cc @AlexWaygood

[charliermarsh]

I can take if you're off for the day, Alex, up to you.

[AlexWaygood]

I can take if you're off for the day, Alex, up to you.

Yes please, thanks!

[namurphy]

Thank you for the quick response, and for respecting work-life balance! Admittedly the affected users may be limited to physicists who spend too much of their time looking up Unicode tables, so it's not too urgent.

Also I should clarify that ℂℇℊℋℌℍℎℐℑℒℓℕℤΩℨKÅℬℭℯℰℱℹℴ runs ever-so-slightly counter to my usual advice for naming things. 🙃

[charliermarsh]

No prob, I like fixing stuff like this.

[zanieb]

Is there a scientific repository we can add to the ecosystem checks that would catch this?

[namurphy]

Thank you for the amazingly quick bugfix! I'm starting to understand better the difficulties of dealing with Unicode edge cases...

Is there a scientific repository we can add to the ecosystem checks that would catch this?

I found it in PlasmaPy in our notebook on Coulomb logarithms, using nbqa-ruff.