Maintaining symbol keys for data and metadata with JSON mapper

[mostlyobvious]

Recently when working on #507 I've reminded myself how ActiveJob handles hashes serialized to be passed over background job queues. It does not only pass the hash with job arguments, but also some metadata that describes it.

One part of this metadata is _aj_symbol_keys. By memorizing which hash keys were symbols, they're able to convert them back when deserialized.

I think this strategy could work for JSON serializer. If we retain list/map of symbol keys and store it in metadata column, then the mapper would be able to use that metadata on deserialization. There's no need to leak this metadata down to event objects, just keeping it in the SerializedRecoed and below and getting rid of it (metadata.delete(:json_mapper)) when constructing event object.

With this approach we could handle symbols transparently and don't advise event schemas "as a must" for JSON serialization.

WDYT @valscion @joelvh @paneq?

Related:
https://github.com/rails/rails/blob/master/activejob/lib/active_job/arguments.rb#L54
https://github.com/rails/rails/blob/master/activejob/lib/active_job/serializers/symbol_serializer.rb
https://github.com/rails/rails/blob/master/activejob/lib/active_job/serializers.rb

[joelvh]

Keys aren't the only issue -- dates are as well. I forget at the moment how I handled that, but I do think some sort of schema tracking is necessary to be able to convert back from the storage format to the correct data types.

[valscion]

I'm 👎 on this. It feels like a lot of unnecessary effort. If people want schemas, there are better solutions out there, such as protobuf.

Our only concern with JSON was that the plain serialize/deserialize was broken because strings as keys after JSON parsing caused a crash.

I would not use this feature, we would still symbolize all the keys after JSON data was parsed from the database. We would not want to bloat the database with this json mapper data — it could get quite big if the JSON had a lot of keys.

[mostlyobvious]

for the record, an example of dry-struct/dry-types schema: RailsEventStore/ecommerce@13ae349

[mostlyobvious]

It feels like a lot of unnecessary effort. If people want schemas, there are better solutions out there, such as protobuf

I'm leaning towards schemas as well. I might give it a spin though when #572 hits production, just to illustrate the concept.

[valscion]

Thanks for following up @pawelpacana. Looking forward to the mappers pipeline 😀