Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: reset pruned sender numbers on stage drop #11150

Merged
merged 1 commit into from
Oct 2, 2024

Conversation

hai-rise
Copy link
Contributor

We may get over 100% stage progress if we do:

  1. Prune some senders
  2. Drop the sender recovery stage
  3. Rerun the sender recovery stage
INFO ... stage=SenderRecovery checkpoint=4889589 target=19426586 stage_progress=101.19%
...
INFO ... stage=SenderRecovery checkpoint=16606018 target=19426586 stage_progress=177.06%

This is because we count pruned transactions towards progress:

fn stage_checkpoint<Provider>(provider: &Provider) -> Result<EntitiesCheckpoint, StageError>
where
Provider: StatsReader + StaticFileProviderFactory + PruneCheckpointReader,
{
let pruned_entries = provider
.get_prune_checkpoint(PruneSegment::SenderRecovery)?
.and_then(|checkpoint| checkpoint.tx_number)
.unwrap_or_default();
Ok(EntitiesCheckpoint {
// If `TransactionSenders` table was pruned, we will have a number of entries in it not
// matching the actual number of processed transactions. To fix that, we add the
// number of pruned `TransactionSenders` entries.
processed: provider.count_entries::<tables::TransactionSenders>()? as u64 + pruned_entries,

This naive fix resets the pruned numbers when we drop the sender recovery stage. Hope it doesn't break something else 😅.

Copy link
Collaborator

@shekhirin shekhirin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fair, and we need to do the same for other segments

@shekhirin shekhirin added C-bug An unexpected or incorrect behavior A-staged-sync Related to staged sync (pipelines and stages) A-pruning Related to pruning or full node labels Sep 24, 2024
Copy link
Collaborator

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, ty

cc @joshieDo for vis, the other checkpoints we also need to update here @shekhirin ?

should we do this here or as followups?

@mattsse mattsse added this pull request to the merge queue Oct 2, 2024
@mattsse
Copy link
Collaborator

mattsse commented Oct 2, 2024

sorry forgot to pull the trigger on this one

Merged via the queue into paradigmxyz:main with commit c4ce997 Oct 2, 2024
36 checks passed
@hai-rise hai-rise deleted the reset-pruned-senders-on-drop branch October 2, 2024 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-pruning Related to pruning or full node A-staged-sync Related to staged sync (pipelines and stages) C-bug An unexpected or incorrect behavior
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants