Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port changes to sorting & localized sorting #220

Merged
merged 19 commits into from
Nov 17, 2020
Merged

Port changes to sorting & localized sorting #220

merged 19 commits into from
Nov 17, 2020

Conversation

olanti-p
Copy link
Contributor

@olanti-p olanti-p commented Nov 16, 2020

Summary

SUMMARY: I18N "Switch lists in some menus to use localized sorting; improve the sorting on Windows and MacOS (ported from DDA)"

Purpose of change

Port from DDA PRs that switch various menus to use localized sorting (instead of sorting by id, or by blindly comparing strings).
Also port PRs that fix localized sorting on Windows and MacOS.

Describe the solution

Cherry-picked without changes (the single merge conflict was due to different header inclusions, wow) the following PRs from DDA:
CleverRaven#40100
CleverRaven#40131
CleverRaven#44288
CleverRaven#40062
CleverRaven#40041
CleverRaven#40291
CleverRaven#44078
CleverRaven#41104

Describe alternatives you've considered

Looking for a cross-platform library that can:

  1. Convert utf-8 string <-> ascii string <-> wstring
  2. Compare 2 utf-8 strings alphabetically
  3. Format numbers & dates
  4. Do all of the aforementioned without being affected by the current locale / platform.

Right now those are done via locale-dependent standard library or platform-specific crutches, and these solutions are far from reliable. From personal experience:

  1. Running test suite from DDA's master on my Linux results in 2 failed number formatting tests.
  2. Both BN and DDA, when launched on my Windows 10, cannot work with non-ascii paths (conversion from wstring to utf-8 seems to be broken).

However, integrating such library into the codebase is a big project, with all the problems that entails.

Testing

List in "wish for items" is now sorted alphabetically under both Linux and Windows 10 in both English & Russian.
List of character templates is now sorted alphabetically under Linux in both English and Russian, and under Windows 10 in English.
Didn't test the rest of the menus, since the code looks correct.
Didn't test on MacOS or Android, since I don't own the first and can't build for second.

Additional context

akirashirosawa and others added 19 commits November 12, 2020 18:29
Core Foundation functions have names that indicate when you own a returned object:
Object-creation functions that have "Create" embedded in the name;
Object-duplication functions that have "Copy" embedded in the name.
If you own an object, it is your responsibility to relinquish ownership (using CFRelease) when you have finished with it.
On Windows the existing solution for localized comparison seems to do
the wrong thing, but evidence suggests that comparison of wstring should
work.

Convert the strings before comparison on Windows.

At the same time, do the reverse conversion on MacOS (this won't
actually be used anywhere in the current code, but it seemse a good idea
to implement it while we had the experimental data to suggest it was
necessary.
Look for calls which sort strings in a non-localized manner.
These are the ones that were caught by the recently improved check.
The two checks were detecting string types in completely different ways.
Make them both behave the same way, and remove the now-unused approach.
Warn about comparison or sorting of pairs or tuples where one of the
elements is a string.
This doesn't actually catch any more real cases in the current code, but
it future-proofs while I'm thinking about such things.
The results were already being sorted, it makes sense to be consistent
and sort the components too.
sort_case_insensitive_less relied on toupper(), which will not work well
for non-English languages.

Remove it and replace its uses with localized_compare.
Sort the list of item types in the wish menu.  Using a localized sort on
the names as displayed in that menu.
@Coolthulhu
Copy link
Member

I can't really test MacOS myself either. But the tests pass, meaning it at least isn't causing memory corruption.
The only MacOS specific changes here should either cause a test failure or a crash, so passing tests should imply it's working.

@Coolthulhu Coolthulhu merged commit b6749ba into cataclysmbnteam:upload Nov 17, 2020
@olanti-p
Copy link
Contributor Author

olanti-p commented Nov 17, 2020

the tests pass

I don't see travis though. Did it not run, or github just hides it from me for some reason?
Either way, the comments under original PRs confirm the solution works, and I didn't find any more PRs in DDA that deal with localized sorting. If there are any issues left, they're most likely minor

@olanti-p olanti-p deleted the better-i18n branch November 17, 2020 10:11
@Coolthulhu
Copy link
Member

The travis was broken for some reason. After I fixed it, I looked at the history and saw that this PR only failed at Android builds, which are currently bugged for PRs because of #200.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants