Skip to content
This repository has been archived by the owner on Jan 22, 2025. It is now read-only.

Remove deletion of TransactionStatusIndex entries #34023

Merged

Conversation

steviez
Copy link
Contributor

@steviez steviez commented Nov 11, 2023

Problem

These entries are legacy code at this point; however, older release branches (v1.16 and v1.17) require these entries to be present. Also, while it would be nice to clean up these entries immediately, they only occupy a small amount of space so having them linger a little longer isn't a big deal.

Summary of Changes

Hold off on deleting these entries until a later point in time.

Funny enough, we've had some churn here:

This issue was reported in Discord by Brooks who was trying to run v1.17/v1.16 solana-ledger-tool on a ledger that had been created with master.

Copy link

codecov bot commented Nov 11, 2023

Codecov Report

Merging #34023 (beb3aa7) into master (e832765) will increase coverage by 0.0%.
Report is 2 commits behind head on master.
The diff coverage is 0.0%.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #34023   +/-   ##
=======================================
  Coverage    81.9%    81.9%           
=======================================
  Files         819      819           
  Lines      219350   219347    -3     
=======================================
+ Hits       179698   179754   +56     
+ Misses      39652    39593   -59     

@steviez steviez marked this pull request as ready for review November 11, 2023 21:20
@steviez
Copy link
Contributor Author

steviez commented Nov 11, 2023

Hmm, I'm now doubting that I fully understand the problem Brooks encountered after spelling out the calls. My current understanding of what is happening with the tip of master:

  1. cleanup_old_entries() populates the TransactionStatusIndex entries at startup if they're absent

    fn cleanup_old_entries(&self) -> Result<()> {
    if !self.is_primary_access() {
    return Ok(());
    }
    // Initialize TransactionStatusIndexMeta if they are not present already
    if self.transaction_status_index_cf.get(0)?.is_none() {
    self.transaction_status_index_cf
    .put(0, &TransactionStatusIndexMeta::default())?;
    }
    if self.transaction_status_index_cf.get(1)?.is_none() {
    self.transaction_status_index_cf
    .put(1, &TransactionStatusIndexMeta::default())?;
    }

  2. update_highest_primary_index_slot() is called immediately after during startup

    • If the entries are freshly initialized, meta.max_slot will be 0 for both entries which means self.highest_primary_index_slot will remain a None
    • If the entries already existed and had some non-zero max_slot == S, then self.highest_primary_index_slot will be Some(S)
      fn update_highest_primary_index_slot(&self) -> Result<()> {
      let iterator = self.transaction_status_index_cf.iter(IteratorMode::Start)?;
      let mut highest_primary_index_slot = None;
      for (_, data) in iterator {
      let meta: TransactionStatusIndexMeta = deserialize(&data).unwrap();
      if highest_primary_index_slot.is_none()
      || highest_primary_index_slot.is_some_and(|slot| slot < meta.max_slot)
      {
      highest_primary_index_slot = Some(meta.max_slot);
      }
      }
      if highest_primary_index_slot.is_some_and(|slot| slot != 0) {
      self.set_highest_primary_index_slot(highest_primary_index_slot);
      } else {
      self.db.set_clean_slot_0(true);
      }
      Ok(())
      }
  3. maybe_cleanup_highest_primary_index_slot() will get called once BlockstoreCleanupService starts actually cleaning via set_max_expired_slot()

    fn maybe_cleanup_highest_primary_index_slot(&self, oldest_slot: Slot) -> Result<()> {
    let mut w_highest_primary_index_slot = self.highest_primary_index_slot.write().unwrap();
    if let Some(highest_primary_index_slot) = *w_highest_primary_index_slot {
    if oldest_slot > highest_primary_index_slot {
    *w_highest_primary_index_slot = None;
    self.transaction_status_index_cf.delete(0)?;
    self.transaction_status_index_cf.delete(1)?;
    }
    }
    if w_highest_primary_index_slot.is_none() {
    self.db.set_clean_slot_0(true);
    }
    Ok(())
    }

But, in 3. above, self.highest_primary_index_slot should be a None, so the if statement that deletes the TransactionStatusIndex entries (the one that this PR modifies) should not run. Thus, those entries should not be getting deleted after the initial creation.

So, as mentioned, it is unclear to me how Brook's blockstore got into the state it was. Will probably revisit Monday once I've had some time to think on it. Regardless, I don't think we should be deleting these entries anywhere so this PR is likely good to go

These entries are legacy code at this point; however, older release
branches require these entries to be present. Also, while it would be
nice to clean up these entries immediately, they only occupy a small
amount of space so having them linger a little longer isn't a big deal.
@steviez steviez force-pushed the bstore_primary_index_back_compat branch from e9a0c1d to beb3aa7 Compare November 27, 2023 18:58
Copy link
Contributor Author

@steviez steviez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rebased in order to re-run CI with the latest incase something might have changed

ledger/src/blockstore.rs Show resolved Hide resolved
Copy link
Contributor

@CriesofCarrots CriesofCarrots left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay!

ledger/src/blockstore.rs Show resolved Hide resolved
@steviez steviez merged commit 70cab76 into solana-labs:master Dec 13, 2023
17 checks passed
@steviez steviez deleted the bstore_primary_index_back_compat branch December 13, 2023 04:53
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants