Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TrieDB: use im::OrdMap to reduce memory consumption #1141

Merged
merged 2 commits into from
Mar 20, 2023
Merged

TrieDB: use im::OrdMap to reduce memory consumption #1141

merged 2 commits into from
Mar 20, 2023

Conversation

kvinwang
Copy link
Collaborator

@kvinwang kvinwang commented Jan 31, 2023

The im::HashMap has a really heavy memory overhead. Let's use im::OrdMap to reduce memory usage.
image

Drawback

im::OrdMap is a bit slower than im::HashMap. It took about 57 minutes to sync from block height 0 to 900000 vs 52 minutes for im::HashMap.

@kvinwang kvinwang requested review from jasl and h4x3rotab January 31, 2023 05:34
Base automatically changed from fix-mq-abi to master January 31, 2023 08:02
Copy link
Collaborator

@jasl jasl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, have you compare saving checkpoint of these data structures?

@kvinwang
Copy link
Collaborator Author

kvinwang commented Jan 31, 2023

LGTM, have you compare saving checkpoint of these data structures?

No, I haven't. There is no significant difference in the early blocks. In order to make a meaningful comparison, I think it needs to sync to recent blocks which make big-size checkpoints.

@jasl Could you build a test version and ask @doyleguo to test it?

@jasl
Copy link
Collaborator

jasl commented Jan 31, 2023

LGTM, have you compare saving checkpoint of these data structures?

No, I haven't. There is no significant difference in the early blocks. In order to make a meaningful comparison, I think it needs to sync to recent blocks which make big-size checkpoints.

@jasl Could you build a test version and ask @doyleguo to test it?

No Problem, this is PRuntime V2 right? BTW, does this need to back port to v0 to replace std::Hash one?

@kvinwang
Copy link
Collaborator Author

kvinwang commented Jan 31, 2023

this is PRuntime V2 right? BTW, does this need to back port to v0 to replace std::Hash one?

Yep, this is V2. For v0 we already replaced std::HashMap with std::BTreeMap in #1121. It doesn't need to backport this PR.

im::OrdMap is almost the same as std::BTreeMap on memory consumption.

@kvinwang kvinwang changed the title TrieDB: use im::OrdMap instead of im::HashMap as backend TrieDB: use im::OrdMap to reduce memory consumption Jan 31, 2023
Copy link
Contributor

@h4x3rotab h4x3rotab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@kvinwang kvinwang merged commit ac65c00 into master Mar 20, 2023
@kvinwang kvinwang deleted the ordmap branch March 20, 2023 06:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants