Support for struct ISerializer?

[AlgorithmsAreCool]

Was toying with a ref struct ISerializer that would write to a Span<byte>. But I would need to avoid boxing the serializer instance. Do you think that changing ISerialize<T> to:

public interface ISerialize<T>
{
    void Serialize<TSerializer>(T value, ref TSerializer serializer) where TSerializer : ISerializer;
}

Would be worth it to allow this use case? (Might also allow the JIT to devirt the ISerializer method calls 🤔)

[agocke]

If you have a ref struct serializer, I think you would also need the new allows ref struct support in .NET 9. Otherwise, you couldn't substitute a ref struct in as the generic argument. Unfortunately, I want to support the latest LTS, which is .NET 8.

Also, I think I want to support async, which will mean no ref variables of any kind.

Lastly, I did previously have that API and I did look at the codegen. Not a good win, surprisingly. In almost all cases I tested it was improving performance by maybe 1%, but it was making the binary significantly larger in Native AOT due to the specialized codegen, and it was increasing compile times.

I'll think about adding such an API, but right now I'm not inclined to fork the API if I want to support both async and sync.

[AlgorithmsAreCool]

Yeah i know there are complex tradeoffs to consider here 😅. I think i can work around it with some fancier footwork and serializing into a IBufferWriter. I'll report back if I hit a wall

[AlgorithmsAreCool]

@agocke Are the methods on ISerializer<T> like SerializeBool and SerializeByte used byte the source generator? I was loosely expecting that the source gen would call the "correct" methods on the ISerializer instance, but looking at the generated code, it looks like it is sending all the fields through ISerializeType.SerializeField.

example gen:

partial record struct PrimaryObjectHeader : Serde.ISerialize<Fulcrum.Rio.Services.Vector.PrimaryObjectHeader>
{
    void ISerialize<Rio.Services.Vector.PrimaryObjectHeader>.Serialize(Rio.Services.Vector.PrimaryObjectHeader value, ISerializer serializer)
    {
        var _l_typeInfo = PrimaryObjectHeaderSerdeTypeInfo.TypeInfo;
        var type = serializer.SerializeType(_l_typeInfo);
        type.SerializeField<Rio.Services.Vector.ObjectType, Rio.Services.Vector.ObjectTypeWrap>(_l_typeInfo, 0, value.ObjectType);
        type.SerializeField<Rio.Services.Vector.ObjectFlags, Rio.Services.Vector.ObjectFlagsWrap>(_l_typeInfo, 1, value.ObjectFlags);
        type.SerializeField<uint, UInt32Wrap>(_l_typeInfo, 2, value.ObjectPayloadLength);
        type.SerializeField<ushort, UInt16Wrap>(_l_typeInfo, 3, value.ExtentLength);
        type.SerializeField<long, Int64Wrap>(_l_typeInfo, 4, value.NextExtentOffset);
        type.End();
    }
}

Since my ISerializeType class is going to have to route all the calls itself, why bother defining the complex ISerializer<T> interface?

[agocke]

Also, here's an example of a custom type using one of the Serialize calls directly: https://github.com/dn-vm/dnvm/blob/d1f6dd29b26655297672e0ce0e805e617f3b5941/src/dnvm/SerdeWraps/SemVersionSerdeWrap.cs#L26

In that case, the type is simply represented as a string, so it can directly express how it wants to be serialized using the serializer APIs. ISerializeType would only be used if the type is represented as some form of composite data structure in the underlying serializer format.

[AlgorithmsAreCool]

Oh, i'm dumb. I didn't realize i could just do this:

public void SerializeField<T, U>(TypeInfo typeInfo, int index, T value, U serialize) 
    where U : ISerialize<T>
{
    serialize.Serialize(value, serializer);
}

Thats super easy.

[AlgorithmsAreCool]

I've been reapproaching this and had a couple follow-up questions

1. I noticed built-in types like Int32Wrap and StringWrap both have state, but they seem to only be pass-through types that never use their state variables. Is this intentional?

2. After studying the generated code, I am a little unclear on the purpose of the IDeserializeVisitor<T>. I think that it allows another layer of integration for custom deserialization. For example, I could provide a custom IDeserializeVisitor<MyCustomModel> that can read Json into a 3rd party type while still leveraging a serde compatible serializer. But it doesn't seem to be directed needed by Serde to deserialize normal objects via the generated code. Is that correct? And if so does it make sense to be included in by default in IDeserializer?

[agocke]

Great questions. In summary, this area has been in flux as I introduce some new abstractions in order to simplify the common case and improve performance.

I noticed built-in types like Int32Wrap and StringWrap both have state, but they seem to only be pass-through types that never use their state variables. Is this intentional?

Pure legacy code. I should remove the state.

I think that it allows another layer of integration for custom deserialization.

yeah that’s right. The big advantage is that it allows the implementor to decide how to deserialize after having seen the data first. So for example, let’s say you wanted to deserialize a json payload into a json type that can hold lists, dictionaries, numbers, etc. The problem is that you don’t know what’s in the payload before you start reading it. The Visitor allows serde to call in saying “give me the next item”, and then the visitor calls one of the visitor methods to retrieve the next item, based on what it sees in the payload. For example, if it sees ‘{‘ it will call VisitDictionary.

In contrast, the ISerdeInfo stuff is great for types that know what they want to deserialize, but doesn’t work well if you don’t know what comes next.

[AlgorithmsAreCool]

Pure legacy code. I should remove the state.

Cool, I understood correctly =D

...let’s say you wanted to deserialize a json payload into a json type that can hold lists, dictionaries, numbers, etc. The problem is that you don’t know what’s in the payload before you start reading it...

So this seems to be a dichotomy in use cases.

On the one hand there is a kind of "deterministic" deserialization that is served by ISerdeInfo where the type shape is known and the type shape "drives" the (de)serializer field by field.

OTOH, there is the "non-deterministic" or data-driven approach that the visitor pattern serves and the deserializer drives the object construction flow lexeme by lexeme.

If I understand this correctly, it is very insightful design. But my (unsolicited) gut opinion tells me they are separate concerns that might be a bit crowded in the same interface. But at least I grok the design now!