Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix long sync cache #18514

Merged
merged 7 commits into from
Aug 26, 2024
Merged

fix long sync cache #18514

merged 7 commits into from
Aug 26, 2024

Conversation

almogdepaz
Copy link
Contributor

@almogdepaz almogdepaz commented Aug 22, 2024

Purpose:

only warmup the cache at the forkpoint

this pr pulls the cache warmup up to the sync_from_fork_point call and avoids calling the warmup on each batch

Current Behavior:

we currently dont add blocks that we already know during reorg long sync that means we need to load these blocks each time from the db

New Behavior:

add blocks that are already known so we dont have to warmup the cache each time, this is ok since we clean the cache up to BLOCKS_CACHE_SIZE after each batch

Testing Notes:

since we use add_block in the tests for big reorg we need to manually warmup to the forkpoint before looping and adding each block

@github-actions github-actions bot added the merge_conflict Branch has conflicts that prevent merge to main label Aug 22, 2024
Copy link
Contributor

This pull request has conflicts, please resolve those before we can evaluate the pull request.

@almogdepaz almogdepaz added Blockchain team Issues tagged for Blockchain team to work on full_node Fixed Required label for PR that categorizes merge commit message as "Fixed" for changelog and removed merge_conflict Branch has conflicts that prevent merge to main labels Aug 22, 2024
@almogdepaz almogdepaz marked this pull request as ready for review August 22, 2024 18:02
@almogdepaz almogdepaz requested a review from a team as a code owner August 22, 2024 18:02
Copy link
Contributor

File Coverage Missing Lines
chia/full_node/full_node.py 38.5% lines 1063-1064, 1068-1073
Total Missing Coverage
20 lines 8 lines 60%

@almogdepaz
Copy link
Contributor Author

almogdepaz commented Aug 22, 2024

our reorg tests use a chain the forks beofre the second sub epoch, that means the fork point will always be 0, coverage failing since we dont test reorg cases where fork point is not 0

Copy link
Contributor

@arvidn arvidn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@arvidn
Copy link
Contributor

arvidn commented Aug 23, 2024

the tests I made intended to reorg from a non-zero height, but I think there's a quantizing effect of reorging from the nearest epoch, or something like that. Is that right?
Would the tests need to for from a greater height?

@almogdepaz
Copy link
Contributor Author

the tests I made intended to reorg from a non-zero height, but I think there's a quantizing effect of reorging from the nearest epoch, or something like that. Is that right? Would the tests need to for from a greater height?

in long sync we find the fork point using only the sub_epoch_summary list (from the wp) we compare the sub_epoch_summary lists and we start syncing 2 sub epochs to avoid forking attacks in the last common sub epoch

i think your change to test-cache should be more then enough but if we changed all the forked chains in the test-cache we might be missing the forkpoint == 0 (unless theres an explicit test for that)

@pmaslana
Copy link
Contributor

pmaslana commented Aug 23, 2024

@arvidn @almogdepaz Are we good to remove the coverage-diff label? I am assuming so due to the comment regarding the fork point, but I wanted to verify first.

@arvidn
Copy link
Contributor

arvidn commented Aug 24, 2024

@pmaslana actually, I'm working on a patch against this PR to extend the test coverage

@arvidn
Copy link
Contributor

arvidn commented Aug 26, 2024

@pmaslana extending the tests will take a bit longer than I thought. I think it's OK to land this first. I will follow up with my additional tests.

@pmaslana pmaslana merged commit 483ec60 into main Aug 26, 2024
372 of 373 checks passed
@pmaslana pmaslana deleted the fix_long_sync_cache branch August 26, 2024 12:48
@altendky altendky mentioned this pull request Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Blockchain team Issues tagged for Blockchain team to work on Fixed Required label for PR that categorizes merge commit message as "Fixed" for changelog full_node
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants