Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(lib/trie): cache inserted trie node Merkle values #2881

Conversation

qdm12
Copy link
Contributor

@qdm12 qdm12 commented Oct 7, 2022

Changes

  • Production code changes
  • Fixes all tests
  • Change delta tracking to use node pointers to improve performance
  • Remove dirty bool field
  • Create issue for child tries delta tracking

Tests

go test -tags integration github.com/ChainSafe/gossamer/lib/trie/...

Issues

Fixes #2854 and unblocks lazy loading impl

Primary Reviewer

@qdm12 qdm12 changed the base branch from development to qdm12/trie/fix-deleted-keys October 7, 2022 15:25
@qdm12 qdm12 force-pushed the qdm12/trie/fix-deleted-keys branch 3 times, most recently from 67676eb to 146fbbb Compare October 12, 2022 12:05
@qdm12 qdm12 force-pushed the qdm12/trie/fix-deleted-keys branch from 1ee69d6 to e0c6446 Compare October 18, 2022 07:30
@qdm12 qdm12 force-pushed the qdm12/trie/cache-inserted-merkle-values branch from 4a60ec4 to fa2ebd4 Compare October 24, 2022 15:22
@qdm12 qdm12 changed the base branch from qdm12/trie/fix-deleted-keys to qdm12/trie/tracking-struct October 24, 2022 15:24
@qdm12 qdm12 force-pushed the qdm12/trie/cache-inserted-merkle-values branch 2 times, most recently from 17926fc to 48229bd Compare October 27, 2022 15:50
@qdm12 qdm12 force-pushed the qdm12/trie/tracking-struct branch 2 times, most recently from 64e183d to 2389072 Compare November 2, 2022 16:06
@qdm12 qdm12 force-pushed the qdm12/trie/cache-inserted-merkle-values branch from 48229bd to 3d5fcef Compare November 2, 2022 16:23
@qdm12 qdm12 force-pushed the qdm12/trie/cache-inserted-merkle-values branch from 3d5fcef to 29a2415 Compare November 2, 2022 16:25
@qdm12
Copy link
Contributor Author

qdm12 commented Nov 2, 2022

This is simply not possible in terms of cpu performance.
Re-calculating the merkle value (scale encoding + blake2b) of N nodes when inserting a node at depth N in the trie is too expensive to be a working solution. This also highlights the need for:

  • lazy-hashing: only compute the state root of the trie when needed, and not on every insertion (already the case in our code)
  • lazy loading needs a write cache, meaning we keep deltas in a in-memory part of the trie without encoding/hashing/writing to disk, and only do the root hash computation + write of deltas when needed (i.e. at the end of handling a block).

Closing this for now as a failed but taughtful experiment.

@qdm12 qdm12 closed this Nov 2, 2022
@qdm12 qdm12 deleted the qdm12/trie/cache-inserted-merkle-values branch November 2, 2022 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Faster online pruning with cached trie inserted/modified merkle values
1 participant