string.ToUpperInvariant("μ") gives different results on different platforms

[anthony-c-martin]

Description

We're using string.ToUpperInvariant() to normalize a JSON blob for case-insensitive hashing, and noticed that if the string contains a "μ", the hash is different across different platforms.

Results from the following GitHub Actions build agents:

• windows-latest: string.ToUpperInvariant("μ") -> μ
• ubuntu-latest: string.ToUpperInvariant("μ") -> M
• macos-latest: string.ToUpperInvariant("μ") -> M

I can set up a repo with GitHub Actions to repro this if that's helpful here.

Configuration

• .NET version: net5.0
• OS versions (from GitHub docs):
  • windows-latest: https://github.com/actions/virtual-environments/blob/releases/win19/20210608/images/win/Windows2019-Readme.md
  • ubuntu-latest: https://github.com/actions/virtual-environments/blob/releases/ubuntu20/20210614/images/linux/Ubuntu2004-README.md
  • macos-latest: https://github.com/actions/virtual-environments/blob/releases/macOS-10.15/20210612/images/macos/macos-10.15-Readme.md

Regression?

Unsure - only noticed when exercising this functionality for the first time.

Other information

N/A

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ufcpp]

https://docs.microsoft.com/en-us/dotnet/core/compatibility/globalization/5.0/icu-globalization-api
https://docs.microsoft.com/en-us/dotnet/standard/globalization-localization/globalization-icu

<ItemGroup>
  <RuntimeHostConfigurationOption Include="System.Globalization.UseNls" Value="true" />
</ItemGroup>

using System;

// Micro Sign
Console.WriteLine(char.ToUpperInvariant('\u00B5')); // U+039C in ICU, U+00B5 in NLS

// Greek Small Letter Mu
Console.WriteLine(char.ToUpperInvariant('\u03BC')); // U+039C in both ICU and NLS

[ghost]

Tagging subscribers to this area: @tarekgh, @safern
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

We're using string.ToUpperInvariant() to normalize a JSON blob for case-insensitive hashing, and noticed that if the string contains a "μ", the hash is different across different platforms.

Results from the following GitHub Actions build agents:

• windows-latest: string.ToUpperInvariant("μ") -> μ
• ubuntu-latest: string.ToUpperInvariant("μ") -> M
• macos-latest: string.ToUpperInvariant("μ") -> M

I can set up a repo with GitHub Actions to repro this if that's helpful here.

Configuration

• .NET version: net5.0
• OS versions (from GitHub docs):
  • windows-latest: https://github.com/actions/virtual-environments/blob/releases/win19/20210608/images/win/Windows2019-Readme.md
  • ubuntu-latest: https://github.com/actions/virtual-environments/blob/releases/ubuntu20/20210614/images/linux/Ubuntu2004-README.md
  • macos-latest: https://github.com/actions/virtual-environments/blob/releases/macOS-10.15/20210612/images/macos/macos-10.15-Readme.md

Regression?

Unsure - only noticed when exercising this functionality for the first time.

Other information

N/A

Author:	anthony-c-martin
Assignees:	-
Labels:	area-System.Globalization, untriaged
Milestone:	-

[GrabYourPitchforks]

@tarekgh Here's another example of a use case that could benefit from a first-party "case folding" API, as ToUpperInvariant isn't guaranteed to remain stable across .NET releases or across versions of ICU.

[tarekgh]

@anthony-c-martin you are using Windows Server 2019 which by default not having the ICU library we depend on. That is why this you may experience the difference.

You can try including the ICU in your app when running on Windows Server 2019. You can do that by

<ItemGroup>
  <PackageReference Include="Microsoft.ICU.ICU4C.Runtime" Version="68.2.0.6" />
  <RuntimeHostConfigurationOption Include="System.Globalization.AppLocalIcu" Value="68.2" />
</ItemGroup>

This should help.

@GrabYourPitchforks

Here's another example of a use case that could benefit from a first-party "case folding" API, as ToUpperInvariant isn't guaranteed to remain stable across .NET releases or across versions of ICU.

Moving forward when users using ICU users should experience consistent results. This is not specific to the casing operation only but for collation operations too. I am not opposing the idea; I am just trying to have the casing and the collation are consistent even for Invariant. That is why I prefer if we expose the case folding as its own APIs and not tie it to invariant.

I am closing this issue as we already having issue #20674 tracking the request. @anthony-c-martin feel free to send any more questions if you have any. Thanks for your report.

[majastrz]

Does the RuntimeHostConfigurationOption workaround also work with the single-file self-contained apps?

[tarekgh]

Does the RuntimeHostConfigurationOption workaround also work with the single-file self-contained apps?

Yes, it does.

[anthony-c-martin]

@tarekgh - do you have any guidance for authors of shared libraries targeting .NET standard? In our scenario, the logic which relies upon .ToUpperInvariant() lives in a nuget package which targets netstandard2.0, and is consumed by various .NET 5 applications.

We can go and fix the applications we own which depend on this package, but is there a way we can guarantee that any consumer of this package will see consistent behavior?

[tarekgh]

@anthony-c-martin could you tell more about your scenario. you mentioned you are hashing JSON blob. Where this blob came from and how you are using the hashing of this blob?

[anthony-c-martin]

@anthony-c-martin could you tell more about your scenario. you mentioned you are hashing JSON blob. Where this blob came from and how you are using the hashing of this blob?

It's a user-defined document (example here) which is usually either user-authored, or programmatically generated. The API which handles this document is case-insensitive, so we want a means of generating a hash to compare two documents for (case-insensitive) equivalence - mostly for telemetry. We also own some cross-platform client tools which work with templates, so we'd like to ensure we can reliably generate the same hash client-side.

The hashing logic we run on the JSON document roughly looks like:

1. Strip non-semantic whitespace
2. Normalize object ordering
3. Uppercase everything (using string.ToUpperInvariant()
4. Calculate a hash with a standard hashing algorithm

[tarekgh]

Thanks @anthony-c-martin for sharing the details. If you want to always guarantee the exact same casing/hashing between the server and client, you'll need to do one of the following:

• Always do the hashing on the server side to ensure the hashing is centralized and ensure consistency.
• Use ICU package and require apps consuming your library to reference this specific version of ICU.
• Encapsulate the casing mechanism in your library so you'll be isolated from any running environment change. There is the library https://github.com/dotnet/corefxlab/tree/archive/src/System.Text.CaseFolding which you may use it to do so. you may copy the code (as no development happening on this library), or you can reference the NuGet package from the feed https://dnceng.visualstudio.com/public/_packaging?_a=package&feed=dotnet-experimental&package=System.Text.CaseFolding&version=0.1.2-alpha.21059.1&protocolType=NuGet. We are considering adding these APIs in the future releases of the .NET.