Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

disable state history log file compression #33

Merged
merged 1 commit into from
Apr 16, 2024

Conversation

spoonincode
Copy link
Member

I would like to disable ship's log file compression.

Compressing and writing the log is done synchronously from the main thread so I feel like these operations should be as quick as possible. (I can think of some complex ways to compress on another thread, but would suggest not adding more complexity). This change writes non-compressed zlib streams in to the log. The benefit of doing it this way is that the logs remain backward compatible with older versions -- nothing about the log format changed. Some small marginal overhead remains due to the zlib framing and checksumming.

The performance benefit from this change is substantial, with the initial state write being reduced from 12m30s to 6m20s. A significant remainder of the time is spent in some seemingly grossly inefficient ostream churn so I suspect when the writing is optimized further the zlib compression will come out as having more than a 2x overhead.

Of course, the size of the logs will be larger. In this case the initial state increased from 5.2GB to 44GB. A user who cares about space usage has other options available to them such as filesystem compression (which can use better compression algos like zstd). Or, we could potentially after a configuration item for compression vs no compression?

@spoonincode spoonincode changed the title RFC: disable state history log file compression disable state history log file compression Apr 16, 2024
@spoonincode spoonincode merged commit bef48a2 into main Apr 16, 2024
36 checks passed
@spoonincode spoonincode deleted the disable_ship_compression branch April 16, 2024 15:08
@ericpassmore
Copy link
Contributor

ericpassmore commented Apr 17, 2024

Note:start
group: NOTICE
category: INTERNALS
summary: Disable state history log file compression to improve performance. Users who would like to compress files can use other methods outside of nodeos.
Note:end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants