Ephemetoot may die when encountering utf8 encoded toots

[billalbertson]

Adding "PYTHONIOENCODING=utf-8" fixes the error. Error and post error with utf-8 setting below.

bills-grimoire:ephemetoot> python3 ephemetoot.py --test                          
This is a test run...
Fetching account details...
Checking 1314 toots...
Traceback (most recent call last):
  File "ephemetoot.py", line 80, in checkToots
    + toot.created_at.strftime("%d %b %Y")
UnicodeEncodeError: 'ascii' codec can't encode character '\u274c' in position 0: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ephemetoot.py", line 139, in <module>
    checkToots(timeline)
  File "ephemetoot.py", line 121, in checkToots
    checkToots(next_batch, deleted_count)
  File "ephemetoot.py", line 121, in checkToots
    checkToots(next_batch, deleted_count)
  File "ephemetoot.py", line 121, in checkToots
    checkToots(next_batch, deleted_count)
  [Previous line repeated 4 more times]
  File "ephemetoot.py", line 111, in checkToots
    print("\U0001f6d1 Unknown ERROR deleting toot - " + str(toot.id))
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f6d1' in position 0: ordinal not in range(128)

bills-grimoire:ephemetoot> PYTHONIOENCODING=utf-8 python3 ephemetoot.py --test 
This is a test run...
Fetching account details...
Checking 1314 toots...
<snip>
Test run. This would have removed 1044 toots.

[hughrun]

Hmm, weird, Python3 should read and write strings using utf-8 by default. Can you tell me more about your environment, @billalbertson? What OS and Python version are you running?

[billalbertson]

Running OpenBSD 6.5, with the python-3.6.8p0 package installed. I can verify that directly when I get home.

…

On July 22, 2019 12:20:41 AM PDT, Hugh Rundle ***@***.***> wrote: Hmm, weird, Python3 should read and write strings using utf-8 by default. Can you tell me more about your environment, @billalbertson? What OS and Python version are you running? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #11 (comment)

___ Bill Albertson [email protected]

[hughrun]

Hey @billalbertson I had another look at this and it looks like this should be more or less resolved as of Python 3.7

Essentially the problem is that Python likes to cooperate with other tools and libraries, and prior to v3.7 will always use the locale settings in the host OS. LIke several other *nix systems, OpenBSD uses US ASCII as the default locale character encoding. As of Python 3.7 there is now a setting called PYTHONCOERCECLOCALE which defaults to being on and coercing the locale value to UTF-8. If you're interested in the background there's a big explanation in Python PEP 538 which led to the change.

So I guess there are three possible solutions for anyone encountering this:

1. Use the PYTHONIOENCODING=utf-8 env setting when running the script, as you have described
2. Set the OS locale to UTF-8 in your ~/.profile file (e.g. export LC_CTYPE=en_US.UTF-8) - though this might have unintended consequences
3. Upgrade to Python 3.7

I probably should add something in the docs to this effect.

[billalbertson]

Thank you for the additional information and follow up! Love ephemetoot...

…

On July 22, 2019 3:07:30 PM PDT, Hugh Rundle ***@***.***> wrote: Hey @billalbertson I had another look at this and it looks like this should be more or less resolved as of Python 3.7 Essentially the problem is that Python likes to cooperate with other tools and libraries, and prior to v3.7 will always use the locale settings in the host OS. LIke several other *nix systems, OpenBSD [uses US ASCII as the default](https://man.openbsd.org/locale.1) locale character encoding. As of Python 3.7 there is now a setting called [PYTHONCOERCECLOCALE](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONCOERCECLOCALE) which defaults to being on and coercing the locale value to UTF-8. If you're interested in the background there's a big explanation in [Python PEP 538](https://www.python.org/dev/peps/pep-0538/) which led to the change. So I guess there are three possible solutions for anyone encountering this: 1. Use the `PYTHONIOENCODING=utf-8` env setting when running the script, as you have described 2. Set the OS locale to UTF-8 in your ~/.profile file (e.g. `export LC_CTYPE=en_US.UTF-8`) - though this might have unintended consequences 3. Upgrade to Python 3.7 I probably should add something in the docs to this effect. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #11 (comment)

___ Bill Albertson [email protected]

[hughrun]

Thanks for bringing it to my attention!