How is synchronization designed?

[renyuneyun]

I'm trying to develop a local-first Solid app (actually to convert an open-source app) with synchronization to Solid Pod, and wanted to find the appropriate tools. Remembered Soukai, found it mentioned synchronization through history but no details; then came to Umai, found this function about synchronization.

I roughly get the idea, but not completely. To sum it up, I have the following understands and questions:

1. The synchronization functionality is designed by the App, not provided by Soukai, right?
   • Of course some feature of Soukai is needed to support it, especially history. But Soukai's API is not enough for synchronization, right?
2. Is this synchronization function complete and correct? I'm not entirely persuaded that concurrent changes will always be dealt with correctly, especially when involving both adding and deleting.
   • For example, what if Remote has deleted a recipe, and Local has added more content into that recipe? Is it simply discarded, or will there be dangling data?
3. Is Soukai's history and/or Umai's synchronization implemented as CRDT, or something else? How is the performance (or should I worry about performance, for something like a note-taking / diary app)?
4. If I'm going to design synchronization, what other design considerations are needed apart from handling all data through Soukai-Solid?

Any hints or comments are appreciated.

[NoelDeMartin]

Hey, I think you're the first person I've seen besides me trying to make a local-first Solid App, so I'm looking forward to see how it goes. Do keep me updated :).

Before I get into the details, I talked a bit about this in a recent presentation I gave at the Solid Symposium, so if you haven't watched I suggest that you check it out: Solid CRDTs in practice.

The synchronization functionality is designed by the App, not provided by Soukai, right?

Yes, correct. What Soukai does is keep track of all the changes to a single model. That means that whenever any changes are done to a single model, besides changing the data of the model itself; it'll create the associated operations. That's mostly what I talk about in the presentation.

What Umai does, is that I have two instances for each model. One is the "local" instance, using an IndexedDBEngine; and another is the "remote" instance, using a SolidEngine. A brief explanation of what this CloudService does is:

1. Pull remote changes into remote models.
2. Merge local and remote models (this is what that synchronizeModels function is doing).
3. Save all dirty models (dirty means that something has changed since the last save). Local models will be updated in IndexedDB, and remote models will be updated in the Solid POD. They should all be the same now since their operations were merged in step 2.

That's the gist of it, but there are many things in that code that won't work for any application, it's quite specific to Umai. For example, it also handles file uploads, and it doesn't take care of merging changes in containers, etc. I'm now working on making another Solid app local-first, and I started coding this algorithm from scratch. Eventually, I want to make it generic enough that it can work with any container/document structure, but at the moment it's a bit specific to my apps.

Is this synchronization function complete and correct? I'm not entirely persuaded that concurrent changes will always be dealt with correctly, especially when involving both adding and deleting.

As I mention in the previous question, as of now it's not complete and correct for any data structure. But I guess it should work to give you an idea on how you'd do it in your app.

When deleting, the way it works in Umai is that it's leaving a tombstone behind. So when a recipe is deleted, the document still exists but all it contains is <#it> a crdt:Tombstone. So if the app is trying to synchronize a deleted model, it'll be deleted locally as well and the local changes will be discarded (you can see that in the instanceof Tombstone part).

This is just a decision I made for my app, but I guess you can choose to handle it differenly. For example, you can use soft deletes (which is what I do with the local models before they are synchronized). Instead of converting the model into a tombstone, you just add a crdt:deletedAt property and ignore that model in the UI (or show it in the trash or something).

Is Soukai's history and/or Umai's synchronization implemented as CRDT, or something else? How is the performance (or should I worry about performance, for something like a note-taking / diary app)?

Yes, it is a CRDT, look at the presentation for more details.

The performance is probably not great :/. Mostly because there is a lot of data overhead, but I don't think the synchronization code takes a lot to run. Soukai is not super well-optimized for performance either, because none of my apps have needed it so far. But I don't see why it couldn't be optimized if necessary, the only problem I'm not sure how I'd solve is the data overhead.

For a note-taking / diary app I guess it depends how you implement it. If you want to synchronize changes on every keystroke, it'll definitely be a problem. With CRDTs, it depends which is your leaf data node. In Umai, for example, even though it's a CRDT recipe instructions are the whole leaf node (instead of each character in each instruction). In practice, each RDF triple is a leaf node. So if you try to change an instruction in two devices, once they are merged only one will remain. I think this depends on what's important for your app.

If I'm going to design synchronization, what other design considerations are needed apart from handling all data through Soukai-Solid?

I think I mentioned most relevant things. Maybe the only important quirk I didn't mention is that initially, models are created without any history. Only after their first change, are operations created (I did this to reduce data overhead for recipes that are created but never edited). The only problem with this, is if the first change happens in two devices and you need to resolve the conflict. In that case, the "inception operations" (the operations with the initial data of the model) will have different ids. At the moment, I'm just sorting them alphabetically and removing the duplicates but that's a bit fickle. I haven't found any issues in practice though.