HashWithIndifferentAccess breaks deserialization when using JSON serializer

[jasonjho]

With the latest 0.34.0 release, the recommendation when using the json serializer is to use HashWithIndifferentAccess for event data to deal with symbolization.

class MyEvent < RailsEventStore::Event
  def data
    ActiveSupport::HashWithIndifferentAccess.new(super)
  end
end

However, I suspect this might break when using the ActiveJobScheduler due to the fact that to_h is called on the serialized record before being passed to the active job handler as seen here: (https://github.com/RailsEventStore/rails_event_store/blob/master/rails_event_store/lib/rails_event_store/active_job_scheduler.rb#L4).

Based on how the DefaultMapper serializes the event (https://github.com/RailsEventStore/rails_event_store/blob/master/ruby_event_store/lib/ruby_event_store/mappers/default.rb#L15), this would yield something like:

@data="--- !ruby/hash:ActiveSupport::HashWithIndifferentAccess\nfoo: :bar\n"

which would break any attempt to deserialize it downstream.

Is there potentially something I'm missing here?

[mostlyobvious]

Correct, the recommendation for serializer: JSON in DefaultMapper is to pair it with HashWithIndifferentAccess.new on your event class. To be precise — the recommendation is to wrap event data on read only. There is no passing of ActiveSupport::HashWithIndifferentAccess.new as data: into event class constructor.

Otherwise you'd have to be disciplined in using only string keys on data.

Another option could be relying on event schemas but I haven't explored that topic in depth with regard to JSON deserialization and string/symbol issue (maybe @andrzejsliwa has?). The related video on that topic could be this one.

Speaking of ActiveJobScheduler:

• as you pointed out, it operates on SerializedRecord

• SerializedRecord is a simplification of an event, the data and metadata payload is already converted to strings and suitable for passing further to repository or, in this case, background job processor

• event handler (in form of background job) is expected to deserialize such SerializedRecord back to your event object, in docs there's an example of AsyncHandler module which calls Rails.configuration.event_store.deserialize(payload) for you

  class SendOrderEmail < ActiveJob::Base
    prepend RailsEventStore::AsyncHandler
  end

• to_h is called on SerializedRecord so that we match method signature of https://github.com/RailsEventStore/rails_event_store/blob/master/ruby_event_store/lib/ruby_event_store/client.rb#L237-L239

• besides RailsEventStore turning event into SerializedRecord and back to event there's also serialization happening in background job processor — when perform_later is called, this depends on configured async job adapter, what we pass here is already a Hash with values being String

Does this answer your doubts? If you've already hit an issue, I'd appreciate backtrace to investigate further.

Btw. what background processor (or active job adapter) are you relying on?

[mostlyobvious]

I think we might be a bit too optimistic in RailsEventStore in expecting that what we get from background job as a payload is always a Hash with symbolized keys.

In reality that part depends on background processor choice typically that could be a Hash with stringified keys.

In fact I have following code in one of the projects:

def perform(arg)
  fact = event_store.deserialize(arg.symbolize_keys)
  ...
end

I need to actually verify this, thanks!

[mostlyobvious]

However, https://github.com/RailsEventStore/rails_event_store/blob/v0.34.0/ruby_event_store/lib/ruby_event_store/mappers/default.rb#L15 will call the read-only data wrapper method from the Event class which gets passed to the JSON serializer dumps command.

You're right on this one. So in the end with that recommendation we'll be serializing ActiveSupport::HashWithIndifferentAccess, not plain Hash into JSON.

But this produces the following serialized string that looks like: --- !ruby/hash:ActiveSupport::HashWithIndifferentAccess\nfoo: :bar\n

I must disagree here. This is a result of:

irb(main):018:0> YAML.dump(ActiveSupport::HashWithIndifferentAccess.new(foo: :bar))
=> "--- !ruby/hash:ActiveSupport::HashWithIndifferentAccess\nfoo: :bar\n"

Not exactly the result of:

irb(main):019:0> JSON.dump(ActiveSupport::HashWithIndifferentAccess.new(foo: :bar))
=> "{\"foo\":\"bar\"}"

If this is the value of data within the serialized_record, then upon deserialize it will break the corresponding JSON.load here https://github.com/RailsEventStore/rails_event_store/blob/v0.34.0/ruby_event_store/lib/ruby_event_store/mappers/default.rb#L25

We should be using the same mapper for interacting for repositories and background jobs. So if you configure RailsEventStore to use JSON:

Rails.application.configure do
  config.to_prepare do
    Rails.configuration.event_store = RailsEventStore::Client.new(
      mapper: RubyEventStore::Mappers::Default.new(
        serializer: JSON
      )
    )
  end
end

then this is used both to convert events before database storage and on scheduling them for handling in background job workers (as an async event handlers)

I've put simplified code to illustrate that flow: https://gist.github.com/pawelpacana/7610921c937aabeec1e107153c8d4429

[jasonjho]

You are right. I realized this was user error on my part since I didn't specify the JSON serializer in the rspec where this was happening (though the application itself works fine). Sorry for bringing you down this rabbit hole, but having this clarified was worth it. Thank you! Will close the issue.

…

On Thu, Nov 29, 2018 at 6:54 AM Paweł Pacana ***@***.***> wrote: However, https://github.com/RailsEventStore/rails_event_store/blob/v0.34.0/ruby_event_store/lib/ruby_event_store/mappers/default.rb#L15 will call the read-only data wrapper method from the Event class which gets passed to the JSON serializer dumps command. You're right on this one. So in the end with that recommendation we'll be serializing ActiveSupport::HashWithIndifferentAccess, not plain Hash into JSON. But this produces the following serialized string that looks like: --- !ruby/hash:ActiveSupport::HashWithIndifferentAccess\nfoo: :bar\n I must disagree here. This is a result of: irb(main):018:0> YAML.dump(ActiveSupport::HashWithIndifferentAccess.new(foo: :bar)) => "--- !ruby/hash:ActiveSupport::HashWithIndifferentAccess\nfoo: :bar\n" Not exactly the result of: irb(main):019:0> JSON.dump(ActiveSupport::HashWithIndifferentAccess.new(foo: :bar)) => "{\"foo\":\"bar\"}" If this is the value of data within the serialized_record, then upon deserialize it will break the corresponding JSON.load here https://github.com/RailsEventStore/rails_event_store/blob/v0.34.0/ruby_event_store/lib/ruby_event_store/mappers/default.rb#L25 We should be using the same mapper for interacting for repositories and background jobs. So if you configure RailsEventStore to use JSON: Rails.application.configure do config.to_prepare do Rails.configuration.event_store = RailsEventStore::Client.new( mapper: RubyEventStore::Mappers::Default.new( serializer: JSON ) ) end end then this is used both <https://github.com/RailsEventStore/rails_event_store/blob/master/ruby_event_store/lib/ruby_event_store/client.rb#L27-L34> to convert events before database storage and on scheduling them for handling in background job workers (as an async event handlers) I've put simplified code to illustrate that flow: https://gist.github.com/pawelpacana/7610921c937aabeec1e107153c8d4429 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#506 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAYX5mtls0gX0eZBZNwo0Y5APAfoeyrzks5uz8rggaJpZM4Y4jLY> .

[mostlyobvious]

No problem, it helped me isolate related issue (#507) and clarify docs that with JSON not only symbol keys are converted to string but values as well:

JSON.load(JSON.dump(foo: :bar))
=> {"foo"=>"bar"}

[mostlyobvious]

Clarification: a690bc8