Ensure that NVDA sets Python locale to a valid Windows locale when initializing language.

[lukaszgo1]

Link to issue number:

Fix-up of #12214
Follow up to #12250
Fixes root cause of #12160

Summary of the issue:

After NVDA sets its interface language it also attempts to set Python locale to the current language. Before PR #12214 we were just passing the current language code to locale.setlocale and catching all possible exceptions. Unfortunately Python setlocale is pretty useless on Windows as it tries to normalize the locale name according to the Unix standards (see Python issue 37945. When we were using wxPython older than 4.1 this was generally not an issue (except for some specific locales under a particular configurations such as one described in #12160 ) since wxPython was setting locale to a valid one so the problem remained unnoticed. This broke with wxPython 4.1 and the fix was attempted in #12214 but it still was setting Python locale to Unix locales. This is causing two problems:

• We are Windows only so it is important to set locales to something reasonable
• If the locale that is unknown to Windows is set some versions of CRT assumes that the code page of such locale is Utf8 which causes a crash when using time.localtime (that was the underlying cause of Crash of the latest alpha NVDA builds on a system with Russian locale #12160)

Description of how this pull request fixes the issue:

Since Python locale.setlocale and locale.getlocale cannot be trusted on Windows the approach I took avoids all the normalization they offer by creating a valid Windows locale string using getLocaleInfoEx in a form which Python no longer normalizes. This string is then passed to locale.setlocale. The new functionality has pretty solid unit test coverage, previous unit tests for languageHandler are also updated to account for these changes. As suggested during the review this PR also consolidates all NLS constants we use when retrieving various locale related information from Windows into an enum in languageHandler. The old constants are deprecated and would no longer be available in 2022.1.
Specific fix for #12160 from #12250 cannot be removed because even though we're setting only complete locales i.e. with a code page and for languages such as Russian or German_Switzerland this crash would no longer occur NVDA supports some locales such as Hindi for which code page is indeed Utf8 and they would still cause a crash.

Testing strategy:

Manual testing:
On Windows 7, Server 2012, Windows 10 1809 (on which I was previously able to reproduce #12160 ) and latest preview of Windows 11 started NVDA with both specific language set in preferences and with the language set to default. Made sure that when 'user default' is selected appropriate language is chosen and locale (as retrieved by locale.setlocale(locale.LC_ALL)) is set into a locale of a form englishLanguageName_englishCountryName.ANSICodePage. Repeated these tests with some languages known to cause troubles in the past:

• Aragonese - locale set to Windows locale
• German_Switzerland and Hindi - made sure that Crash of the latest alpha NVDA builds on a system with Russian locale #12160 has not been reintroduced

Executed unit tests on all above mentioned versions of Windows made sure they pass.

Known issues with pull request:

None known

Change log entries:

Bug fixes:

• NVDA no longer sets invalid Python locales

Changes for developers:

• LOCALE_SLANGUAGE, LOCALE_SLIST and LOCALE_SLANGDISPLAYNAME are moved to the LOCALE enum in languageHandler. They are still available at the module level but deprecated and would be removed in NVDA 2022.1

Code Review Checklist:

• Pull Request description is up to date.
• Unit tests.
• System (end to end) tests.
• Manual testing.
• User Documentation.
• Change log entry.
• Context sensitive help for GUI changes.
• UX of all users considered:
  • Speech
  • Braille
  • Low Vision
  • Different web browsers

[lukaszgo1]

@seanbudd Given your work on #12214 you may be interested in taking a look at this.

[seanbudd]

Thanks @lukaszgo1 for your impressive work on this.

I've left some notes on code styling, and requested some additional tests, but the logic here looks solid.
Once we can get confirmation that this works for users on other systems, as requested here, I think this can be merged. If this can't be done in a reasonable timeframe, we can move forward to alpha testing.

[lukaszgo1]

@seanbudd I believe all your review comments are now addressed. I've simplified the implementation of setLocale though had to do so differently than you've suggested since after applying your suggestions Linter started to complain about it being too complex. Contrary to my initial belief I also had to keep the code from #12250 - the updated PR description explains in which cases it is necessary. The flip side of this is that it is now much safer to merge this into Alpha.

[seanbudd]

Thanks for implementing the enum and deprecations and pointing out the tests I missed. This looks almost ready to merge.

I've requested some further changes to setLocale to make it easier to visually parse. I'm not sure if this sort of code-styling is helpful for vision impaired developers - if it is detrimental I am happy to ignore it.

[lukaszgo1]

@seanbudd All addressed.

[seanbudd]

Thanks @lukaszgo1, this change will be much appreciated