Replicate operations in topo order #442
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was
linked to
issues
Jul 9, 2023
adzialocha
approved these changes
Jul 9, 2023
4 tasks
adzialocha
added a commit
that referenced
this pull request
Jul 14, 2023
* development: (23 commits) Implement `dialer` behaviour (#444) Sort expected results in strategy tests Update CHANGELOG Replicate operations in topo order (#442) Maintain sorted operation indexes (#438) Use fork of `asynchronous-codec` (#440) Ingest check for duplicate entries (#439) Reverse lookup for pinned relations in dependency task (#434) Remove unnecessary exact version pinning in Cargo.toml Make `TaskInput` an enum and other minor clean ups in materialiser (#429) Use `libp2p` `v0.52.0` (#425) Fix race condition when check for existing view ids was too early (#420) Reduce logging verbosity CI: Temporary workaround for Rust compiler bug (#417) Fix early document view insertion (#413) Handle duplicate document view insertions (#410) Decouple p2panda's authentication data types from libp2p's (#408) Remove dead_code attribute in lib Integrate replication manager with networking stack (#387) Implement naive replication protocol (#380) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We want to validate all operations which arrive at a node via replication before persisting them to the node and performing materialization. In order to do this the document they are part of needs to be in it's most recent state before the in-coming operation was published. This can be achieved if entries are ordered appropriately before they are sent to a remote node in replication request responses.
When preparing entries response messages in
LogHeighStrategythey should be grouped by document id and then ordered bysorted_indexof their operations. This way a remote can simply usepublishto ingest all arriving entries as they arrive in their topologically sorted order and can be directly validated and added to the document.When consuming entries in
Ingestwe should use thepublishmethod provided byp2panda-rs, this already performs all required validation.publishto ingest entries arriving on a node via replicationThese changes mean that we now require that authors don't publish concurrently to the same document with the same key pair, this needs adding to the spec: p2panda/website#284
📋 Checklist
CHANGELOG.md