Dynamically adjust state sync RAM usage #858
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yilunzhang
reviewed
Jun 6, 2022
config/config.go
Outdated
| @@ -461,6 +465,15 @@ func Init() error { | |||
| Parameters.SyncMode = SyncMode | |||
| } | |||
|
|
|||
| if Parameters.SyncStateMaxPeer == 0 { | |||
| syncStateMaxPeer := uint64(float64(memory.TotalMemory() / defaultSyncStateThreadMemory)) | |||
There was a problem hiding this comment.
Why not using uint32() instead of uint64 here?
yilunzhang
reviewed
Jun 6, 2022
| @@ -171,10 +172,14 @@ func (s *stateSync) run() error { | |||
|
|
|||
| func (s *stateSync) startWorkerThread() { | |||
| var workerId uint32 | |||
| peersNum := len(s.peers) | |||
| for workerId = 0; workerId < uint32(peersNum*concurrentSyncRequestPerNeighbor); workerId++ { | |||
| peersNum := uint32(len(s.peers)) | |||
There was a problem hiding this comment.
Are you missing concurrentSyncRequestPerNeighbor here?
yilunzhang
reviewed
Jun 6, 2022
node/stateSync.go
Outdated
| go func(workerId uint32) { | ||
| s.startWorker(s.peers[workerId%uint32(peersNum)]) | ||
| s.startWorker(s.peers[workerId%workerNum]) |
There was a problem hiding this comment.
Should be peersNum just in case workerNum > peersNum in the future
yilunzhang
reviewed
Jun 6, 2022
node/stateSync.go
Outdated
| if workerNum > peersNum { | ||
| workerNum = peersNum | ||
| } | ||
| for workerId = 0; workerId < workerNum*concurrentSyncRequestPerNeighbor; workerId++ { |
There was a problem hiding this comment.
concurrentSyncRequestPerNeighbor should be moved to L175, otherwise we will have more workers than config.Parameters.SyncStateMaxPeer
yilunzhang
reviewed
Jun 6, 2022
config/config.go
Outdated
| @@ -461,6 +465,15 @@ func Init() error { | |||
| Parameters.SyncMode = SyncMode | |||
| } | |||
|
|
|||
| if Parameters.SyncStateMaxPeer == 0 { | |||
| syncStateMaxPeer := uint32(float64(memory.TotalMemory() / defaultSyncStateThreadMemory)) | |||
| Parameters.SyncStateMaxPeer = uint32(syncStateMaxPeer) | |||
There was a problem hiding this comment.
No need to do type conversion here
bufrr
force-pushed
the
statesync-oom-fix
branch
2 times, most recently
from
98bc007 to
01c564c
Compare
yilunzhang
reviewed
Jun 6, 2022
node/stateSync.go
Outdated
| @@ -171,10 +172,14 @@ func (s *stateSync) run() error { | |||
|
|
|||
| func (s *stateSync) startWorkerThread() { | |||
| var workerId uint32 | |||
| peersNum := len(s.peers) | |||
| for workerId = 0; workerId < uint32(peersNum*concurrentSyncRequestPerNeighbor); workerId++ { | |||
| peersNum := uint32(len(s.peers) * concurrentSyncRequestPerNeighbor) | |||
There was a problem hiding this comment.
This will cause L182 to use the wrong peersNum. Try something like this (you might need to fix type)
peersNum := len(s.peers)
workerNum := peersNum * concurrentSyncRequestPerNeighbor
if workerNum > config.Parameters.SyncStateMaxPeer {
workerNum = config.Parameters.SyncStateMaxPeer
}
yilunzhang
reviewed
Jun 6, 2022
config/config.go
Outdated
| @@ -357,6 +360,7 @@ type Configuration struct { | |||
| Mining bool `json:"Mining"` | |||
| MiningDebug bool `json:"MiningDebug"` | |||
| BeneficiaryAddr string `json:"BeneficiaryAddr"` | |||
| SyncStateMaxPeer uint32 `json:"SyncStateMaxPeer"` | |||
There was a problem hiding this comment.
Better rename to SyncStateMaxWorkers because concurrentSyncRequestPerNeighbor might be more than 1 in the future
yilunzhang
reviewed
Jun 6, 2022
| @@ -461,6 +465,15 @@ func Init() error { | |||
| Parameters.SyncMode = SyncMode | |||
| } | |||
|
|
|||
| if Parameters.SyncStateMaxThread == 0 { | |||
| syncStateMaxPeer := uint32(float64(memory.TotalMemory() / defaultSyncStateThreadMemory)) | |||
There was a problem hiding this comment.
L469. 470 and defaultSyncStateMaxPeer still have the old name
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes in this pull request
Explain the changes in this pull request in order to help the project maintainers understand the overall impact of it.
Type (put an
xwhere ever applicable)Checklist
Please put an
xagainst the checkboxes. Write a small comment explaining if itsN/A(not applicable)Extra information
Any extra information related to this pull request.