Protobuf cannot be used with async handlers

[paneq]

because: protocolbuffers/protobuf#4391 we always use YAML and google-protobuf does not work with it right now.

• Figure out
• Mutation testing
• Can we use Protobuf's ANY data-type to serialize metadata?
• Update protobuf tutorial in a documentation
• Update mapper in a documentation (no longer add_metadata)

[paneq]

hmmm, but we already have repository.event_to_serialized_record which we can serialize it and repository.serialized_record_to_event which can deserialize it. The only problem is that we would now require something to be included/extended in Async handler classes.

Hmmm, actually right now we require YAML.load, in first line of background job class. From documentation:

class SendOrderEmail < ActiveJob::Base
  def perform(event)
    event = YAML.load(event)
    email = event.data.fetch(:customer_email)
    OrderMailer.notify_customer(email).deliver_now!
  end
end

Perhaps we should require:

class SendOrderEmail < ActiveJob::Base
  def perform(event)
    event = event_store.deserialize(event)
    # ...

That would probably make much more sense as we would be able to change it in any way in the future.

Or maybe:

class SendOrderEmail < ActiveJob::Base
  def perform(event_id, data, metadata, event_type)
    event = event_store.deserialize(data, metadata, event_type)
    # ...

nah... that does not look really good.

[paneq]

yeah, event should be {event_id:, data:, metadata:, event_type:} from SerializedRecord where every value is a String and ActiveJob will know how to handle it thanks to that.

[paneq]

Workaround: protocolbuffers/protobuf#4391 (comment)

[paneq]

It seems that I need to:

• move mapper dependency higher and make it mandatory (because no matter what repo you use we need to know how to serialize/deserialize your events for async handlers.
• pass mapper as dependency to ActiveJobDispatcher#call ? How? ie: call(subscriber, event, mapper) ? Requires changes to RubyEventStore::PubSub::Dispatcher as well
• Calling every Event Repository method would require passing a mapper as it no longer would be a repo dependency.

So that's quite a lot of breaking changes.

[paneq]

Alternatively we expose event_to_serialized_record and serialized_record_to_event as public methods of a Repository and it would be its internal detail whether it delegates to mapper or not.

Maybe it would event make sense to have client.serialize(event) and client.deserialize(string) on top level?

[paneq]

Note's from today's meetup:

• RES serialization - data vs metadata
  • Serializing whole event for sending over network
  • Serializing whole event for sending to background queue
  • Is metadata part of event or not?
  • Is metadata serialized and provided separately to data or not
  • do you need to use RubyEventStore::Event or not
    • Perhaps I should not have exchanged RubyEventStore::Event with Protobuf-event but only RubyEventStore::Event#data content?
      • but then what do you send over the wire? event_id + event_type, + data ?
  • Does protobuf needs to know EVENT_TYPE to deserialize DATA?
    • I think so. Suggest they need to be sent separately
  • When in doubt check GES?
    • https://eventstore.org/docs/http-api/reading-streams/
    • event's data is serialized here
  • worth checking also in Marten https://github.com/JasperFx/marten

[paneq]

hmmm

SerializedRecord#initialize(event_id:, data:, metadata:, event_type:)

vs

RubyEventStore::Event#initialize(event_id: SecureRandom.uuid, metadata: nil, data: nil)

I just realized they are almost identical...

[paneq]

Let's say we went with

client = Client.new(repository: RailsEventStoreActiveRecord::EventRepository.new(
  mapper: RubyEventStore::Mappers::Protobuf.new,
))
client.subscribe(->(ev){@ev = ev}, [ResTesting::OrderCreated])
event = RubyEventStore::Event.new(
  event_id: "f90b8848-e478-47fe-9b4a-9f2a1d53622b",
  data: ResTesting::OrderCreated.new(
    customer_id: 123,
    order_id: "K3THNX9",
))
client.publish_event(event, stream_name: 'test')

then we would also need to reimplement subscribe. In other words, our subscription is tight to class names and not event_types...

[paneq]

In other words, our subscription is tight to class names and not event_types

Here:

      def notify_subscribers(event)
        all_subscribers_for(event.class).each do |subscriber|
          @dispatcher.call(subscriber, event)
        end
      end

[paneq]

https://www.dropbox.com/s/tryu3dyk4jvus32/Screenshot%202018-03-30%2016.21.19.png?dl=0

Interesting to see what bunny (ruby client for rabbitmq does):

require 'bunny'

connection = Bunny.new
connection.start

ch = connection.create_channel
q = ch.queue('', :exclusive => true)
x  = ch.default_exchange

# set up the consumer
q.subscribe(:exclusive => true, :manual_ack => false) do |delivery_info, properties, payload|
  puts properties.content_type # => "application/octet-stream"
  puts properties.priority     # => 8

  puts properties.headers["time"] # => a Time instance

  puts properties.headers["coordinates"]["latitude"] # => 59.35
  puts properties.headers["participants"]            # => 11
  puts properties.headers["venue"]                   # => "Stockholm"
  puts properties.headers["true_field"]              # => true
  puts properties.headers["false_field"]             # => false
  puts properties.headers["nil_field"]               # => nil
  puts properties.headers["ary_field"].inspect       # => ["one", 2.0, 3, [{ "abc" => 123}]]

  puts properties.timestamp      # => a Time instance
  puts properties.type           # => "kinda.checkin"
  puts properties.reply_to       # => "a.sender"
  puts properties.correlation_id # => "r-1"
  puts properties.message_id     # => "m-1"
  puts properties.app_id         # => "bunny.example"

  puts delivery_info.consumer_tag # => a string
  puts delivery_info.redelivered? # => false
  puts delivery_info.delivery_tag # => 1
  puts delivery_info.routing_key  # => server generated queue name prefixed with "amq.gen-"
  puts delivery_info.exchange     # => ""
end

# publishing
x.publish("hello",
          :routing_key => "#{q.name}",
          :app_id      => "bunny.example",
          :priority    => 8,
          :type        => "kinda.checkin",
          # headers table keys can be anything
          :headers     => {
            :coordinates => {
              :latitude  => 59.35,
              :longitude => 18.066667
            },
            :time         => Time.now,
            :participants => 11,
            :venue        => "Stockholm",
            :true_field   => true,
            :false_field  => false,
            :nil_field    => nil,
            :ary_field    => ["one", 2.0, 3, [{"abc" => 123}]]
          },
          :timestamp      => Time.now.to_i,
          :reply_to       => "a.sender",
          :correlation_id => "r-1",
          :message_id     => "m-1")

sleep 1.0
connection.close

[paneq]

The interesting part is that payload (which is like our data) is provided separately from metadata-properties such as message id, correlation id etc (similar to what we put in metadata)

[paneq]

protocolbuffers/protobuf#4455

[paneq]

protocolbuffers/protobuf#4456