Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Init sync optimization #2674

Merged
merged 17 commits into from
Feb 23, 2021
Merged

Init sync optimization #2674

merged 17 commits into from
Feb 23, 2021

Conversation

bmarty
Copy link
Member

@bmarty bmarty commented Jan 14, 2021

Fixes #983

New strategy to handle initial sync: First store the request result into a file, to avoid doing it again in case of crash during treatment. Then perform a partial parsing for big account to avoid too much RAM usage.

Performance change is not significant for small account, it take a bit more time for big account but it consume much less RAM, so there is a better chance to finally finish the sync.

This PR also improve DefaultInitialSyncProgressService and fix a few bug on it

@bmarty bmarty force-pushed the feature/bma/init_sync_poc branch 3 times, most recently from e86d2ce to 7d87618 Compare January 21, 2021 22:03
@bmarty bmarty force-pushed the feature/bma/init_sync_poc branch 3 times, most recently from 9d2453a to 481a8ee Compare February 8, 2021 17:29
@bmarty bmarty marked this pull request as ready for review February 9, 2021 10:26
@bmarty bmarty changed the title Init sync poc Init sync optimization Feb 9, 2021
@bmarty bmarty force-pushed the feature/bma/init_sync_poc branch from 9137da4 to 9e50633 Compare February 16, 2021 11:03
@bmarty bmarty force-pushed the feature/bma/init_sync_poc branch from c42ffd5 to c181f7e Compare February 23, 2021 09:32
@bmarty bmarty merged commit 9db644a into develop Feb 23, 2021
@bmarty bmarty deleted the feature/bma/init_sync_poc branch February 23, 2021 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Stuck on initial sync
3 participants