Update chainbase to tip (reduce memory usage) #1575
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Should we consider adding a validation on |
heifner
approved these changes
Aug 29, 2023
There was a problem hiding this comment.
I think @spoonincode idea of a limit would be good. Otherwise, LGTM.
The new limit for chainbase addressable memory is 8TB, so that would be 8388608mb I believe. |
heifner
requested changes
Aug 29, 2023
heifner
approved these changes
Aug 29, 2023
|
ah wow didn't realize we reduced the max to 8TB |
spoonincode
approved these changes
Aug 29, 2023
| chain_config->state_size = options.at( "chain-state-db-size-mb" ).as<uint64_t>() * 1024 * 1024; | ||
| EOS_ASSERT( chain_config->state_size <= 8ull * 1024 * 1024 * 1024 * 1024, plugin_config_exception, | ||
| "The maximum supported size for the chain state db is 8TiB" ); |
There was a problem hiding this comment.
imo it would be nice for this error message to expressly refer to chain-state-db-size-mb somehow, so the user knows exactly what knob is set too far. But, users can probably figure it out in this case anyways
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This updates the chainbase submodule to include the memory reduction optimization. Description copied below.
Reduce offset_node_base memory usage inside the RBtree of chainbase
This PR will reduce the chainbase memory usage
old implementation:
_parent,_left,_right,_colorconsume 32 bytes together (8 bytes each).new implementation
_parent,_left,_right,_colorconsume 16 bytes (42 + 42 + 42 + 2 bits) together.erased_flagsince _color is a 2 bit signed int.limitations:
_parent,_left&_rightonly have signed 42-bit each (so 41-bit absolute value), shifted by two bits thanks to alignment, implying a maximum of chainbase size of 2^43 = 8TB.offset_node_baseobjects must always be allocated within the same segment due to the 42-bit offset limit. Using a stack variable ofoffset_node_baseis no longer valid (as stack address starts from 0x7fffxxxxxxxx which is too far away from heap addresses).tested results:
using mainnet snapshot snapshot-308122667.bin.tar.gz:
2879913473623136280976(around 20% reduction)[greg 8/21/2023] retested with the current PR version and confirmed the memory savings as reported by
curl -X POST http://127.0.0.1:8888/v1/db_size/getafter loading snapshot.