Browser and trouble with efficiently browsing streams

[mostlyobvious]

Recently we've shipped neat stream and event browser UI for RES. It allows you to navigate through list of streams and list of events in selected stream, finally giving the ability to see particulat event in detail.

Problem: we don't have best database schema to implement stream browsing. Reminder that we have two tables:

rails_event_store/rails_event_store_active_record/lib/rails_event_store_active_record/generators/templates/migration_template.rb

Lines 7 to 27 in 2eb821c

	create_table(:event_store_events_in_streams, force: false) do |t|
	t.string :stream, null: false
	t.integer :position, null: true
	if postgres
	t.references :event, null: false, type: :uuid
	else
	t.references :event, null: false, type: :string
	end
	t.datetime :created_at, null: false
	end
	add_index :event_store_events_in_streams, [:stream, :position], unique: true
	add_index :event_store_events_in_streams, [:created_at]
	add_index :event_store_events_in_streams, [:stream, :event_id], unique: true
	if postgres
	create_table(:event_store_events, id: :uuid, default: 'gen_random_uuid()', force: false) do |t|
	t.string :event_type, null: false
	t.text :metadata
	t.text :data, null: false
	t.datetime :created_at, null: false
	end

In order to get a list of all streams one would have to SELECT DISTINCT stream FROM event_store_events_in_streams. Additionally it would be nice to present all stream at the beginning of this list. The comes the pagination in order to present/navigate though possibly thousands of streams.

What works for small number of streams in dev environement is sort-of-pagination in memory with Enumerable:

rails_event_store/rails_event_store-browser/app/controllers/rails_event_store/browser/streams_controller.rb

Lines 8 to 9 in 2eb821c

	items = event_store.get_all_streams
	items = items.drop_while { |s| !s.name.eql?(position) }.drop(1) unless position.equal?(:head)

rails_event_store/rails_event_store_active_record/lib/rails_event_store_active_record/event_repository_reader.rb

Lines 84 to 88 in 2eb821c

	def get_all_streams
	(["all"] + EventInStream.order(:id).pluck(:stream))
	.uniq
	.map { |name| RubyEventStore::Stream.new(name) }
	end

That however has obvious drawback of being memory-hungry for production use with lots of streams.

Solutions I see to get over it:

0. Ignore and assumme this for dev usage. Likely not going to work 😉

1. Drop stream browsing feature, leaving only event browsing for stream name you know upfront. Works when you know how stream names are chosen in your system and you can recostruct such name yourself.

Morever this is how geteventstore proposes:

One more example from custom audit log implementation:

So in essence there's input field and/or URL in which you could enter stream name in order to land on event browsing for given stream. More effort fo end user but less work on RES side and no one would accidentally OOM-kill app.

2. Implement streams table in form of database view. This would probably allow efficiently paginating like on events. Any drawbacks I failed to see here?

3. Implement event_store_streams. I guess it's the least popular and most troublesome option as it requires migration simillar to what we had in v0.18.0. Not sure if such feature like stream browser ios worth this much effort.

Any ideas?

[paneq]

In order to get a list of all streams one would have to SELECT DISTINCT stream FROM event_store_events_in_streams

Why is that an issue considering that we have compound indexes which include stream at first place which means they can be used for such queries IMHO?

add_index :event_store_events_in_streams, [:stream, :position], unique: true 
add_index :event_store_events_in_streams, [:stream, :event_id], unique: true 

Also, do we really need to get all streams in one query? Could we autocomplete 5-20 streams based on what is currently written in a field?

SELECT DISTINCT stream 
FROM event_store_events_in_streams 
WHERE stream >= 'current input' 
LIMIT 20 
ORDER BY stream ASC

[tomaszwro]

🤔 I'm wondering what is the actual use case where we'd need to be able to see/paginate the list of all streams. Are we sure anyone will really need it?

Searching for streams by stream name is a slightly different thing, but still I'm not sure about its necessity.

I understand that we need to show something on the root browser url - how about just show recent events (regardless of stream).

This would also solve the immediate problem we're having now: browser is hardly usable when there are a lot of streams, because the app crashes every time.

Also, you can see stream names on events list, so it might be that if the user wants to find a certain stream, they might be able to accomplish this by looking for a certain event and then going to its stream.

[paneq]

I'm wondering what is the actual use case where we'd need to be able to see/paginate the list of all streams.

I assume autocomplete on the main page?