Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When taking a snapshot RAM usage increases significantly on 5.0.0-rc3 WAX #2021

Closed
matthewdarwin opened this issue Dec 28, 2023 · 3 comments · Fixed by #2029 or #2040
Closed

When taking a snapshot RAM usage increases significantly on 5.0.0-rc3 WAX #2021

matthewdarwin opened this issue Dec 28, 2023 · 3 comments · Fixed by #2029 or #2040
Assignees
@heifner
Labels
bug Something isn't working 👍 lgtm OCI Work exclusive to OCI team

Comments

@matthewdarwin
Copy link

matthewdarwin commented Dec 28, 2023 •
edited
Loading

When taking a snapshot RAM usage increases significantly on 5.0.0-rc3. Screenshot is from WAX, starting at ~23:10. snapshot is started from producer plugin when the blockchain is up-to-date. Previous snapshots (earlier in the screenshot) were taken while the chain was not up-to-date and we didn't see the same problem.

image

Seems 5.0.0 behaviour is different than earlier versions: #1128

Note also high CPU usage.
@matthewdarwin
Copy link
Author

Going back to nodeos 4.0.x, the problem doesn't happen. (switch happened at 14:00 in below chart)

image

@bhazzard bhazzard added 👍 lgtm bug Something isn't working and removed triage labels Jan 2, 2024
@heifner heifner self-assigned this Jan 2, 2024
@heifner heifner added the OCI Work exclusive to OCI team label Jan 2, 2024
@heifner
Copy link
Member

heifner commented Jan 2, 2024

@matthewdarwin can you provide the 5.0 logs for high ram usage indicated in the graph.

@matthewdarwin
Copy link
Author

Yes, I will send privately.

@ericpassmore ericpassmore linked a pull request Jan 2, 2024 that will close this issue
[5.0] remove per-row tellp() during snapshot creation boosting performance #2026
Merged
@ericpassmore ericpassmore added this to the Leap v5.0.0 milestone Jan 2, 2024
@ericpassmore ericpassmore removed a link to a pull request Jan 2, 2024
[5.0] remove per-row tellp() during snapshot creation boosting performance #2026
Merged
@ericpassmore ericpassmore removed this from the Leap v5.0.0 milestone Jan 2, 2024
heifner added a commit that referenced this issue Jan 3, 2024
@heifner
GH-2021 Pause net_plugin during snapshot write
6d52486
@heifner heifner mentioned this issue Jan 3, 2024
Merged
heifner added a commit that referenced this issue Jan 3, 2024
@heifner
GH-2021 Simplify pause implementation.
c4ed79a
heifner added a commit that referenced this issue Jan 3, 2024
@heifner
GH-2021 Add const
8fe4f3c
@heifner heifner linked a pull request Jan 3, 2024 that will close this issue
[5.0] P2P: Pause net_plugin during snapshot write #2029
Merged
heifner added a commit that referenced this issue Jan 4, 2024
@heifner
Merge pull request #2029 from AntelopeIO/GH-2021-snapshot-ram-5.0
04774eb 
[5.0] P2P: Pause net_plugin during snapshot write
heifner added a commit that referenced this issue Jan 4, 2024
@heifner
Merge remote-tracking branch 'origin/release/5.0' into GH-2021-snapsh…
a6249b5 
…ot-ram-fix
@heifner heifner mentioned this issue Jan 4, 2024
Merged
@heifner heifner linked a pull request Jan 4, 2024 that will close this issue
[5.0 -> main] P2P: Pause net_plugin during snapshot write #2040
Merged
heifner added a commit that referenced this issue Jan 4, 2024
@heifner
Merge pull request #2040 from AntelopeIO/GH-2021-snapshot-ram-fix
9696543 
[5.0 -> main] P2P: Pause net_plugin during snapshot write
This was referenced Sep 12, 2024
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@heifner heifner

Labels
bug Something isn't working 👍 lgtm OCI Work exclusive to OCI team
Projects
None yet
Milestone
No milestone
5 participants
@bhazzard @heifner @ericpassmore @matthewdarwin @enf-ci-bot