[WIP] Draft: Sync Editor React state with Yjs

[dmonad]

Description

This PR enables collaborative editing using the Yjs CRDT library.

⚠️ Protect your data. The current implementation syncs your WordPress posts with a publically available server - for everyone to read ⚠️

Gutenberg is an interesting project for me, and I'm experimenting with making it collaborative. In #17964, we discussed some of the advantages of having Yjs as the collaborative engine for Gutenberg. I offered @epiqueras to create this PR to compare Yjs versus GunDB (another CRDT) for collaborative editing. ⚠️ This PR is a WIP and implements a very basic method to sync the editor state.

Try it out live: https://gutenberg-yjs.now.sh/ - open this link in two browser windows. The content is automatically synced.

Motivation

Yjs is a CRDT library that allows you to share arbitrary state objects efficiently. Unlike GunDB, Yjs is designed as an engine for collaborative (rich)text editors. It supports offline editing, peer-to-peer communication, separate undo/redo stacks for each client, track changes (showing the differences between two states and highlighting them with the color of the creator), and efficient representation of remote selections. Basically, everything you would expect from Google Docs.

Here are a few demos of the editors that Yjs already supports: https://yjs-demos.now.sh/
I would love to add Gutenberg to that list.

Types of changes

I hooked Yjs to the React editor state in the /playground. When an editor block changes, it currently overwrites the complete block content, instead of applying the differences. This is basically the same syncing approach as described in #17964. Therefore it should allow for a fair comparison. But Yjs also allows to apply differences to the text object and is better suited to enable multiple users to work on the same paragraph.

TODO

• Calculate the changes between two different editor states and apply the differences to the Yjs state. This will allow multiple users to work on the same paragraph.
• Convert text formatting attributes (e.g. <bold>..</bold>) to Yjs formats on text objects. This will allow for multiple users to make concurrent formatting changes on overlapping ranges.
• Enable Yjs Undo/Redo in Gutenberg. A local undo should not affect changes from remote clients. Yjs already supports having separate undo/redo stacks for each client, so we only need to hook it up to the keybindings and, if there are any, editor buttons.
• Show remote cursor locations. Remote cursor locations are represented as special markers on text objects. We need a method to render remote cursors and selections in Gutenberg.

[epiqueras]

This is awesome.

So right now this PR has the same edit conflict resolution granularity as #17964, but yeah, we are in a better place to integrate RichText using the Y.Text type.

We need to create a binding for it like the ones for the other rich text editors.

There are two ways I see we can do this:

• Refactor RichText to expose its internal format and edit actions so that we can translate them into something suitable for Y.Text.prototype.{insert,delete,format,applyDelta}.
• Diff the serialized HTML exposed by RichText every time it changes and use that to build suitable format and edit actions for Y.Text.prototype.{insert,delete,format,applyDelta}.

@dmonad I'm guessing that diffing something like bold formatting won't be straightforward as we would have to look for and extract the inserted wrapping tags. @ellatrix how hard would it be to expose RichText's internal data structures instead?

[gziolo]

I created a label to make it easier to track all work related to Collaborative Editing:
https://github.com/WordPress/gutenberg/issues?utf8=%E2%9C%93&q=label%3A%22%5BFeature%5D+Collaborative+Editing%22+

Thank you for opening this PR @dmonad, it's great to see that you started exploring options to integrate your library with the block editor. It looks like the approach takes might want totally independent form WordPress 💯

Do you have some thoughts on how likely it would be to create a reliable communication channel which works between two or more peers without the external server? I'm thinking now in the context of WordPress where everything works with just PHP and MySQL installed on the server. I guess, you still would be able to use PHP to sync between peers but in the version shipped in WordPress core having to set up an external server is rather debatable. Obviously, the end solution would be extensible, so plugin authors could provide all types of enhancements on top of that as in your example.

[dmonad]

We need to create a binding for it like the ones for the other rich text editors.

Correct. I think it makes sense to implement the Gutenberg binding in a separate repository similar to the other editor bindings. But I'm open to suggestions.

• Diff the serialized HTML exposed by RichText every time it changes and use that to build suitable format and edit actions for Y.Text.prototype.{insert,delete,format,applyDelta}.

This is how I implemented the ProseMirror binding. The idea here is to transform the structured RichText content to a linear data structure. I.e. transform

<italic>Hello <div font-size="18">world!</div></italic>

to

[
  { text: 'Hello', attrs: { italic: {} } },
  { text: 'world!', attrs: { italic: {}, div: { font-size: '18' } } } 
]

You can compute the edit actions from the diff of the linearized RichText representation. I already implemented the necessary utility functions for another project.
I can use the serialized RichText content and transform it to the above representation. But it would be more efficient to transform the RichText object directly.

@epiqueras Do you think I hook up to the correct data source (the React state)? I see that you implemented your approach in core-data and I'm wondering if I should do the same.

Do you have some thoughts on how likely it would be to create a reliable communication channel which works between two or more peers without the external server? I'm thinking now in the context of WordPress where everything works with just PHP and MySQL installed on the server. I guess, you still would be able to use PHP to sync between peers but in the version shipped in WordPress core having to set up an external server is rather debatable. Obviously, the end solution would be extensible, so plugin authors could provide all types of enhancements on top of that as in your example.

@gziolo In the web, there is always a server involved to create a p2p connection. Even with WebRTC you need a signaling server to establish the connection between two peers. I think that a community could maintain and finance these signaling servers, because they are relatively cheap. While Gun is decentralized, it does use websockets as a communication protocol and you do need to setup Gun servers.

What do you think about the idea to use WebSockets in WordPress to exchange document updates? The WebSockets module really just needs to forward document updates to the peers.

[ellatrix]

This is how I implemented the ProseMirror binding. The idea here is to transform the structured RichText content to a linear data structure. I.e. transform

Yes, internally, the RichText data structure is already flat, where text and formatting is separated.

@ellatrix how hard would it be to expose RichText's internal data structures instead?

I can use the serialized RichText content and transform it to the above representation. But it would be more efficient to transform the RichText object directly.

I think it would be good to expose the internal structure. Again, this was my intention when the structure was first introduced, but we limited it to be internal only because of back compat reasons. I could see somehow having both in the store, especially since the selection state is already there and works only with the flattened state (it's useless for HTML).

@gziolo In the web, there is always a server involved to create a p2p connection. Even with WebRTC you need a signaling server to establish the connection between two peers. I think that a community could maintain and finance these signaling servers, because they are relatively cheap. While Gun is decentralized, it does use websockets as a communication protocol and you do need to setup Gun servers.

I've been exploring WebRTC separately, where Heartbeat is used for signalling. I'm hoping to share a prototype in the form of a plugin and PR soon.

[dmonad]

I've been exploring WebRTC separately, where Heartbeat is used for signalling. I'm hoping to share a prototype in the form of a plugin and PR soon.

@ellatrix I like the idea to use WP heartbeats. Please let me know when you hooked something up. I've build a Yjs+webrtc example before using simple-peer. If you haven't decided on a webrtc library yet, please have a look at it.

WebRTC is a proper and scalable solution to shared editing. The advantage of a WebSocket connection is that it will work behind proxies/firewalls and on devices with a bad connection. The downside of WebSocket is that it will affect WP performance.

I think it would be good to expose the internal structure.
How can we make this happen?

Can you point me to how Gutenberg implements selections? When a remote client manipulates content, I need to adjust the local selections. Can I manipulate selections in Gutenberg?

[gziolo]

I like the idea to use WP heartbeats. Please let me know when you hooked something up. I've build a Yjs+webrtc example before using simple-peer. If you haven't decided on a webrtc library yet, please have a look at it.

Not that we decided anything, in the exploration done over 2 years ago #1877, exactly the library you recommended was used to establish the connection between peers. And their connection was maintained with something like WP Heartbeat. Looking at the code it was a basic REST API exposed to poll and keep the list of active peers.

[epiqueras]

I think it would be good to expose the internal structure. Again, this was my intention when the structure was first introduced, but we limited it to be internal only because of back compat reasons. I could see somehow having both in the store, especially since the selection state is already there and works only with the flattened state (it's useless for HTML).

@ellatrix

Yes, let's do it.

Can you point me to how Gutenberg implements selections? When a remote client manipulates content, I need to adjust the local selections. Can I manipulate selections in Gutenberg?

@dmonad

It's in the block editor reducer.

WebRTC is a proper and scalable solution to shared editing. The advantage of a WebSocket connection is that it will work behind proxies/firewalls and on devices with a bad connection. The downside of WebSocket is that it will affect WP performance.

Ideally we'll use the Heartbeat API to implement a simple STUN server in the WP instance. If that's not enough, we can introduce a flag that allows the WP instance to run a private fallback relay (TURN server), but this shouldn't be on by default as it would negatively affect the front end performance of sites with many editors. @ellatrix whatever WebRTC library we go with has to support this sort of fallback mechanism so that if full p2p is not supported, it can use the relay. As you suggested elsewhere, I think it makes sense to do this whole feature in the following steps:

• Set up private post-specific WebRTC rooms scoped to people who can edit the given post using the WP instance's Heartbeat API as a STUN server. Potentially adding a flag that enables a fallback relay TURN server.
• Make post locking real-time.
• Show active users and their selections.
• Integrate YJS for block property level edit sharing. (Requires successful WebRTC connection.)
• Integrate RichText for granular edit/formatting/selection sharing in text fields. (Requires successful WebRTC connection.)

When a user doesn't successfully connect over WebRTC or the TURN server is not enabled, we can still show them a locked post with a list of active users and their selections, that updates on every Heartbeat. This makes for a nice progressively enhanced experience.

Note that some things that are required for this like exposing the RichText data structures can be done in parallel and would be beneficial outside of this context.

Does this make sense to move forward? Let's start from the top and bundle it in an experiment (#16626) on this repo instead of a plugin so that it's easier to test.

[dmonad]

I agree with the overall plan. For now we can use the public WebSocket instance to work on content-sharing. From my point of view we have three independed projects:

• Create communication layer between the peers that collaborate on the same document (using WebRTC).
• Handle post-locking in WordPress.
• Create a Yjs data binding for Gutenberg.

I will try to understand how post-locking works. I'm not that familiar with WP, so I might need some help to understand the concepts.

Tomorrow I will look into how blocks are serialized, and I will continue working on the Yjs data binding:

• Show active users and their selections.
• Integrate YJS for block property level edit sharing.
• Integrate RichText for granular edit/formatting/selection sharing in text fields.

[epiqueras]

Yes, we could start work on the last 3 tasks, using the public instance until the first task is finished. That would be awesome.

We need to do this at the core-data level so we'll need to focus on the .blocks edit of the edited post entity like I did in #17964. We can cherry pick #17824 which will add the selection to the entity so that we can sync that as well. I guess we'll also need to add a .users list to keep track of other users editing a given post and their selections.

I guess that prior to doing that, we should:

• Implement a YJS type/binding for Gutenberg blocks.
• Implement a YJS type/binding for Gutenberg selections.

The implementation shouldn't break anything if the connection fails to be established as that is when the Heartbeat fallback would take over and we would show a locked post with a list of active users and their polled selections.

[dmonad]

Thanks for pointing me to #17824.

We need to do this at the core-data level so we'll need to focus on the .blocks edit of the edited post entity like I did in #17964.

Working on it 👍

In the other editor bindings, I propagate selections, and the list of active users using the awareness protocol. They are shared directly via the communication layer and are not really part of the document structure. The awareness protocol implements an efficient solution to share awareness states without a central authority.

FYI: We can't use absolute positions for the selection states, because the document can change before other users are able to update their selection. Using absolute positions would result in weird selection behaviors. Instead, selections should be represented as relative positions / ranges on the Yjs document. They automatically adapt to remote changes.

It would be helpful to me if someone could design and implement the user list for Gutenberg. I can hook up the awareness state to the user list.

[epiqueras]

They are shared directly via the communication layer and are not really part of the document structure.

That looks way more efficient, nice!

Instead, selections should be represented as relative positions / ranges on the Yjs document. They automatically adapt to remote changes.

😲 YJS is amazing, nice work!

It would be helpful to me if someone could design and implement the user list for Gutenberg. I can hook up the awareness state to the user list.

@ellatrix is working on that as part of the first few tasks. It would be nice if we collaborate there.

[dmonad]

Thanks @epiqueras :)

@ellatrix is working on that as part of the first few tasks. It would be nice if we collaborate there.

Yeah definitely! PM me if you want to talk about this. I can also show you some existing designs that you can use as a template.

[epiqueras]

@dmonad You should join us in Slack if you can, https://make.wordpress.org/chat/.

The core-editor channel is a great place to sync with people during the weekly meetings or at any time really.

[LionsAd]

While I agree with the overall plan and agree that splitting up the tasks in:

• Transport
• Fallback
• Real-time sync

Is a good idea this is still missing a crucial part as we learned when we implemented Yjs on a Drupal platform:

• While a real time session is active, undo / redo and seeing who changed what is easy, but there comes a time when changes should be persisted to the Wordpress database.

And also when all editors close the session we still would like to be able to see who made which change, which is information that is usually stored just in the Yjs document. (Unless Gutenberg has their own granular history and I missed that feature).

The solution that we came up with was that we persist the Yjs document from time to time (it does not need to be revisioned as the history grows over time) and store the new published document together with a Yjs state vector into the Yjs document.

That way revisions of the content can be mapped to Yjs documents.

If you don’t need revisions it’s still a good idea to store the original Yjs document which can also bring the state to new clients.

While heartbeat is obviously not ideal, Yjs’ transport agnostic way can still show clients that can’t connect to the p2p session a read only way of what is happening (with some delay obviously)

As Yjs is offline first (also for text editing) even submitting a document could sync changes to other p2p clients.

Just some more food for thought.

[epiqueras]

The solution that we came up with was that we persist the Yjs document from time to time (it does not need to be revisioned as the history grows over time) and store the new published document together with a Yjs state vector into the Yjs document.

I think we can explore this as a last step without compromising any of the current tasks. At the moment, we don't even have single-user revisions working with the block editor, but this is something we definitely want further down the road.

[LionsAd]

Definitely - it indeed is orthogonal to any other tasks. Just saying that at some point a user might want to “publish” the current state and it needs to be added to the overall work plan.

[afercia]

Super interesting! I do realize this is a work in progress so it may depend on something not refined yet but did anyone notice their machine CPU going high while testing https://gutenberg-yjs.now.sh/ ?

After a couple minutes with that page open in Chrome and Firefox, both browsers took ~ 20% of the CPU and my laptop fan started spinning like crazy:

Tested on a Windows 10 Pro laptop with
Intel i7-8550U @ 1.80GHz, 1992 Mhz, 4 Core(s), 8 Logical Processor(s)
16.0 GB RAM

I'd say it's a pretty powerful hardware for a web application. Can't test on a mac right now.

[LionsAd]

Yeah - seeing the same high CPU behavior on a Mac. We don't have this problem with other editors using Yjs [so it's not a structural bug] nor does it happen with Gutenberg standalone [so it's a new thing], so I assume it's an implementation detail / bug of the prototype, which can (and should) be fixed.

Thanks for testing and reporting!

[epiqueras]

I think it's because this prototype diffs every top level block and instead of figuring out how each changed block changed and making the granular modifications, it just replaces the block and all its children entirely which can get expensive depending on how YJS handles it.

[dmonad]

I updated the PR and integrated the current approach in core-data. I believe the current implementation will work nicely in a WebRTC environment. Peers initialize their data after they synced with another peer. This will prevent the issue that new peers overwrite the shared state with the last-saved WordPress state (I noticed that in #17964 the editor eventually overwrites the shared state with the current Wordpress state if you reload the window often - resulting in a loss of content).

Thanks for reporting @afercia. I reduced the performance problems with this update. The diffs don't seem to have any significant performance impact. The problem was that every change will trigger a change in the shared data, which again triggers an update in the editor state with the current state of the document (hence an unnecessary update to the editor state).

The past few days I have been experimenting a lot with the editor. I really like its modular package system 💯

As a next step, I will transform the HTML representation of RichText to the delta format so I can use it in Y.Text. It is not necessary to include the delta library - but I do use their representation in Y.Text. I'm aware that it would be more performant to transform Gutenbergs RichText format directly to a delta, but I'm not sure if that is possible at the moment. So this method will yield quicker results.

[epiqueras]

Nice work!

I'm aware that it would be more performant to transform Gutenbergs RichText format directly to a delta

@ellatrix is there an easy way for us to expose this for this PR?

[justinr1234]

Where did this end up? Thanks, looks really promising!

[dmonad]

Hi @justinr1234 , this project is not forgotten! I'm still working on it but I have been caught up in other projects. In the meantime I have resolved some blockers:

• Now that Yjs v13 has been tested in the wild I finally published a stable Yjs release and converted most of the plugins for v12 to v13.
• I published a y-webrtc connector that we can use in WordPress (WP plugin for signaling still needs to be implemented)
• We published a webinar about y-webrtc and I think we brought up some good points why this is a good fit for Gutenberg https://www.tag1consulting.com/blog/peer-peer-collaborative-editing-using-yjs-webrtc

[justinr1234]

@dmonad great! Is there a plan for how to get this finished for Gutenberg? Can we get a todo list of sorts together? Perhaps with a solid plan we can have multiple people working on it at once. A truly collaborative Gutenberg would be amazing. Even if getting into Wordpress core is a longer stretch goal, having a roadmap to get it for the editor itself (for usage outside Wordpress) would be phenomenal!

[dmonad]

@epiqueras I will find you next week in the Slack channel and maybe we can discuss the necessary steps together.

Here are just a few points I'm thinking about:
• A WordPress plugin that handles signaling. y-webrtc implements a minimal signaling approach that we could easily port to WordPress.
• Access management. y-webrtc encrypts communication over the signaling server (that means you don't have to trust the signaling server). We could implement this similarly to @epiqueras proposal using a shared secret - the server forwards a shared secret to the clients that have access to the shared document.
• Shared cursors in Gutenberg.
• Enable shared editing using a feature flag.

[epiqueras]

Sounds like a great plan!

[justinr1234]

@dmonad how did your meeting with @epiqueras go? Is there new information? Thanks, I greatly appreciate this effort!

[justinr1234]

Hey @dmonad just following up as I’m very excited about this!

[dmonad]

Hey @justinr1234 , sorry for keeping you waiting. I had a nice chat with @epiqueras and it seems that they would be interested in merging this if this feature is enabled under an experimental feature flag and if we encrypt the data using a shared secret. For now, we can use the public signalling servers (they only forward encrypted signalling information). This will be good enough to get this feature out there and have people playing with collaborative editing in Gutenberg.

Furthermore, I'd like to implement signalling as a WordPress plugin that you could re-use to make other WordPress plugins / applications collaborative. At a minimum, we also need to implement presence (who is currently working on a document) and shared cursors in Gutenberg.

While the current implementation looks half-done, we haven't solved all syncing issues yet. Users reported several bugs, and we currently have no method to coordinate what happens when two users hit "save" at the same time. There are many details that still need to be worked out.

In one of the Tag1 interviews we talked about why a lot of people (us included) use Google Docs - or something similar - to write articles, and use the CMS only to format and publish it. In order to make Gutenberg a real alternative to collaborative workflows using Google Docs, I'd also like to bring change-history (revisions, where changes are highlighted by the user who made them) and comments to Gutenberg. You can see change-history already working on yjs.dev.

But I also want to be realistic about the cost that it takes to develop and maintain collaborative editing in Gutenberg. We need funding to bring this project to a final stage. I found sponsors for the other Yjs editor plugins, and I hope to find sponsors for this addition too.

[justinr1234]

@dmonad great info! So the next steps are to fix some of the bugs and get it behind a feature flag for the basic collab editing. Then after that focus on getting financial support to develop it into a full fledged collaborative editing experience?

Is there a way we can define how more people can get involved to help? Perhaps breaking work out into smaller chunks would allow different people to work on this? Is there a possibility of this being some sort of plugin with its own repo where issues can be created and collaborated on?

[dmonad]

@dmonad great info! So the next steps are to fix some of the bugs and get it behind a feature flag for the basic collab editing. Then after that focus on getting financial support to develop it into a full fledged collaborative editing experience?

That sounds reasonable.

Is there a way we can define how more people can get involved to help?
Any help is appreciated! At this time, I believe the best way forward is to merge the changes directly into Gutenberg. We can collaborate in this Gutenberg fork and merge the changes directly into this PR. Step 1 is to put this behind a feature flag - which shouldn't be too hard. Let me know if you want to help out. Later, we also need someone who is familiar with Wordpress to design a signaling plugin.

I just came up with a task-list of the issues that need to be resolved. If needed, I'll describe them better in a separate ticket in https://github.com/dmonad/gutenberg.

• Update project to use y-webrtc @dmonad
• Remove visual clues that say that the document is locked - *
• Put the shared editing addition behind a feature flag - *
• Find/generate a secret token that is only known by peers that have read/write access. The secret token will be used to encrypt webrtc signaling data - *
• Identify and describe shared editing issues (report them here https://github.com/dmonad/gutenberg - not in the official Gutenberg project).
• Optimize state change algorithm and fix sync issues - @dmonad

[paaljoachim]

Hey Kevin @dmonad
Can we get a status update?
Thanks!

[dmonad]

Hi @paaljoachim ,

I have two issues with this.
• The Gutenberg team has made it clear that collaborative editing needs to hold off until phase 3. So I don't think I will get support on the cursors implementation. Even though it would have made sense to integrate it now so Gutenberg can make use of the collaboration-ready undo manager (undo/redo operations from the local user shouldn't affect remote changes).
• I'm still looking for funding to continue on this project. I'd really like to be the driving force behind this, but I can't stem this on my own.

FYI: this is already working very well. I setup a test instance on https://gutenberg.yjs.dev/wp-login.php test / test

Unless I find funding we probably need to hold off until phase 3 begins..

[dmonad]

Somebody just pinged me about instructions on how to test the demo site.

• Go to https://gutenberg.yjs.dev/wp-login.php and login with the credentials test / test
• Click on "Posts" and edit one of the existing Posts or create a new one
• Open the same post in another window.
• The post is collaborative: Multiple users can edit the same document at the same time & document content is automatically synchronized 🥳

[dmonad]

[dmonad]

At this point I also want to mention an informative blog post by publishpress where we talk about our plans of integrating Yjs into Gutenberg: https://publishpress.com/blog/yjs/

[dmonad]

Closed in favor of #23129 - a collab implementation also based on Yjs, but with presence + cursor support and much more!