Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

significant memory consuming during cluster migration #3938

Closed
BorysTheDev opened this issue Oct 16, 2024 · 1 comment · Fixed by #4017
Closed

significant memory consuming during cluster migration #3938

BorysTheDev opened this issue Oct 16, 2024 · 1 comment · Fixed by #4017
Assignees
Labels
bug Something isn't working

Comments

@BorysTheDev
Copy link
Contributor

BorysTheDev commented Oct 16, 2024

given:

two migration processes from one node with a replica to another 2 nodes
the size of the data 2GB
migration for all slots
before migration source node memory consumption is 2.2GB
problem: during migration memory consumption sometimes increases up to 3.6 GB (not every time, also have seen 2.8GB and 2.4GB). Without replication, the maximum memory consumption for this scenario I've seen is 2.8GB.

We must investigate why we have such memory consumption during migration and reduce it. Reproducibility rate 50%.
#3939 reproduce the problem

@BorysTheDev
Copy link
Contributor Author

During migration we do next steps:

RestoreStreamer::WriteEntry() where we serialize the object and create a string with data
JournalWriter::Write() where we do copy of the string
JournalStreamer::Write() where we do one more copy and caching data for sending
If the serialized object is quite big and we send data not fast the size of JournalStreamer::pending_buf_ can be significantrly increased

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants