System.Text.Json: JsonStringEnumConverter ignores its JsonNamingPolicy during deserialization.

[dbc2]

In System.Text.Json in .Net Core 3.1, I can serialize an enum using a naming policy, however when I attempt to deserialize the same enum with a naming policy, the policy is ignored and an exception is thrown when the naming policy does anything beyond modifying the case of the enum value names.

The issue can be reproduced as follows. Define the following simple snake case naming policy:

public class SnakeCaseNamingPolicy : JsonNamingPolicy
{
    public override string ConvertName(string name)
    {
        return string.Concat(name.Select((x, i) => i > 0 && char.IsUpper(x) ? "_" + Char.ToLowerInvariant(x).ToString() : Char.ToLowerInvariant(x).ToString()));
    }
}

And the following enum:

public enum TestEnum
{
    ValueZero,
    ValueOne,
    ValueTwo,
    ValueThree
}

Then attempting to round-trip values of TestEnum will fail during deserialization:

var values = (TestEnum [])Enum.GetValues(typeof(TestEnum));

var options = new JsonSerializerOptions
{
    Converters = { new JsonStringEnumConverter(new SnakeCaseNamingPolicy()) },
    WriteIndented = true,
};

var json = JsonSerializer.Serialize(values, options);

Console.WriteLine("Serialized {0}:", values);
Console.WriteLine(json);

var valuesIn = JsonSerializer.Deserialize<TestEnum []>(json, options); // Exception thrown here!

Assert.IsTrue(valuesIn.SequenceEqual(values));

The enum values are serialized with the naming policy successfully:

[
  "value_zero",
  "value_one",
  "value_two",
  "value_three"
]

But deserialization throws this exception:

Failed with unhandled exception: 
System.Text.Json.JsonException: The JSON value could not be converted to TestEnum[]. Path: $[0] | LineNumber: 1 | BytePositionInLine: 14.
   at System.Text.Json.ThrowHelper.ThrowJsonException()
   at System.Text.Json.Serialization.Converters.JsonConverterEnum`1.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)
   at System.Text.Json.JsonPropertyInfoNotNullable`4.OnReadEnumerable(ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.JsonPropertyInfo.ReadEnumerable(JsonTokenType tokenType, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.JsonPropertyInfo.Read(JsonTokenType tokenType, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack)
   at System.Text.Json.JsonSerializer.ReadCore(Type returnType, JsonSerializerOptions options, Utf8JsonReader& reader)
   at System.Text.Json.JsonSerializer.Deserialize(String json, Type returnType, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)

The problem seems to be that JsonConverterEnum<T>.Read() completely ignores the specified _namingPolicy and simply calls Enum.TryParse():

            // Try parsing case sensitive first
            string? enumString = reader.GetString();
            if (!Enum.TryParse(enumString, out T value)
                && !Enum.TryParse(enumString, ignoreCase: true, out value))
            {
                ThrowHelper.ThrowJsonException();
                return default;
            }
            return value;

A close reading of the documentation here seems to imply that this is an intended limitation:

Reading is case insensitive. Writing can be customized by using a JsonNamingPolicy.

However the inconsistency and inability to round-trip with the same settings is an odd and unexpected restriction. Newtonsoft does not have this limitation -- it can round-trip enum values even when using SnakeCaseNamingStrategy.

Demo fiddle here showing that Json.NET can round-trip enums using SnakeCaseNamingStrategy while System.Text.Json cannot do so with SnakeCaseNamingPolicy.

[NoneGiven]

#1197 is related in that it concerns questionable assumptions about naming policy made by these predefined converters. Of note, if #782 (which is api-approved) is implemented, then this issue will become a bug when using a built-in naming policy rather than just a custom one.

[CodeBlanch]

Anyone looking for a workaround I have a JsonConverter named JsonStringEnumMemberConverter in Macross.Json.Extensions package which fixes this issue and then adds in support for EnumMemberAttribute & Nullable enums. A modified version of dotnet/corefx#41648, basically.

[Bukran]

JsonStringEnumMemberConverter

Thanks a lot! I was stumbling against deserialization.

[Macromullet]

I don't see how this can be addressed with the current API shape, as JsonNamingPolicy.ConvertName is potentially a one-way function. For instance, if you chose HashFunction(name) as the implementation, it's not possible to convert it back. The only way to support this would be to define something like WPF does with converters, where you have Convert and ConvertBack, but that seems to be functionality outside the scope of what System.Text.Json should do.

[layomia]

from @liul in #41138:

runtime/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Value/EnumConverter.cs

Line 317 in a5b491c

if (!Enum.TryParse(enumString, out T value)

Use the obtained string value directly here. But why not use _namingPolicy'?

like:

        internal override T ReadWithQuotes(ref Utf8JsonReader reader)
        {
            string? enumString = reader.GetString();
            
            var v =  _namingPolicy == null ? enumString : FormatEnumValueToString(enumString, encoder);
             
            // Try parsing case sensitive first
            if (!Enum.TryParse(v, out T value)
                && !Enum.TryParse(v, ignoreCase: true, out value))
            {
                ThrowHelper.ThrowJsonException();
            }

            return value;
        }

[2m0nd]

Maybe create cache with source enum value names? And restore source name if exist namingPolicy?
#42302

[avonheimburg]

I don't see how this can be addressed with the current API shape, as JsonNamingPolicy.ConvertName is potentially a one-way function. For instance, if you chose HashFunction(name) as the implementation, it's not possible to convert it back.

For enums, you could actually convert back even with a one-way function. You would need to convert all possible enum values to their serialized form using JsonNamingPolicy.ConvertName and could then simply look up the value you want to deserialize. The only limitation is that the naming policy actually assigns distinct names to all enum values.

[layomia]

Triage: per my comments in #42302 (review) I do not believe this feature is feasible. We'll close this issue.

As a workaround, users can write custom converters for enums that can support a close set of naming-policy-transformed enum names and honor them on deserialization. @tdykstra let's consider providing an example in the JSON docs.

[AlgorithmsAreCool]

Yes we would need to include flags for consistency and to avoid confusion. I do not believe the perf penalty will be acceptable.

Although I realize there are no good answers, I personally think the current solution of ignoring the naming convention for deserialization is more harmful and confusing than poor/missing Flags support.

Again without numbers to back me up, I would expect it is relatively rare for a web APIs to use Flags encoding. That being said, I realize people are using STJ to store and retrieve locally generated data that might need Flags more commonly.

Perhaps a compromise, provide Flags support with the performance fallback, but hidden behind a experimental/preview environment variable until a better solution can be found?

Also, perhaps the limited space cache can be improved by switching to a simple LRU style cache that will avoid a performance "cliff" when the cache overflows. This might amortize the pain somewhat

[tdykstra]

If I run the Main method of the sample and instead of using the custom converter I use the built-in JsonStringEnumConverter with JsonNamingPolicy.CamelCase, everything works correctly. What am I missing that the custom converter is required for?

using System;
using System.Reflection;
using System.Text.Json;
using System.Text.Json.Serialization;

#nullable enable

namespace EnumConverter
{
    public class Program
    {
        public static void Main()
        {
            JsonSerializerOptions options = new() { Converters = { new JsonStringEnumConverter(JsonNamingPolicy.CamelCase) } };

            // If serialization happens first then we likely can support arbitrary values and flag combinations.
            SerializeThenDeserialize(DayOfWeek.Monday);
            SerializeThenDeserialize(BindingFlags.Instance | BindingFlags.Public);

            // Deserialize a "known" enum value without serializing first.
            BindingFlags value = JsonSerializer.Deserialize<BindingFlags>(@"""declaredOnly, createInstance""", options);
            Console.WriteLine(value);

            // Deserialize an "unknown" enum value without serializing first. Recall Enum.ToString() has specific and deterministic ordering for flags.
            try
            {
                JsonSerializer.Deserialize<BindingFlags>(@"""createInstance, declaredOnly""", options);
            }
            catch (JsonException) { }

            void SerializeThenDeserialize(object enumVal)
            {
                string json = JsonSerializer.Serialize(enumVal, options);
                Console.WriteLine(json);

                enumVal = JsonSerializer.Deserialize(json, enumVal.GetType(), options)!;
                Console.WriteLine(enumVal);
            }
        }
    }
}

[layomia]

@tdykstra I think it's working because we have logic trying case-insensitive matching for enum string values, which works well with PascalCase -> camelCase transformations. Can you try your repro using a policy such as snake_case -

runtime/src/libraries/System.Text.Json/tests/Common/TestClasses/TestClasses.cs

Lines 1887 to 1893 in 66ca474

	public class SimpleSnakeCasePolicy : JsonNamingPolicy
	{
	public override string ConvertName(string name)
	{
	return string.Concat(name.Select((x, i) => i > 0 && char.IsUpper(x) ? "_" + x.ToString() : x.ToString())).ToLower();
	}
	}

?

[tdykstra]

That explains it, thanks! Snake case test cases work only with the custom converter. I'll update the docs.

[AlgorithmsAreCool]

@layomia I am assuming there is a 0% chance of a fix for this making it into .NET 6. Is that assumption correct?

[sabitkondakci]

@layomia I am assuming there is a 0% chance of a fix for this making it into .NET 6. Is that assumption correct?

Will this feature be in .Net6 sooner or later?

[eiriktsarpalis]

Will this feature be in .Net6 sooner or later?

We didn't get around to fixing this in time for .NET 6. It will be considered for .NET 7.

[brianlagunas]

I also need this functionality. Seems silly to me that a naming policy would only apply on serialization and not deserialization. Seems like a massive oversight.

Please fix this.