Skip to content
This repository has been archived by the owner on Jan 22, 2025. It is now read-only.

Begin phasing out Blockstore primary indexes #33370

Closed

Conversation

steviez
Copy link
Contributor

@steviez steviez commented Sep 22, 2023

Problem

The existing codebase still contains remnants from when ledger cleaning was done with multiple primary indexes. This PR starts the process of phasing out a variable primary index.

  • This PR fixes the primary index for writes to 0 while continuing to support both 0 and 1 for reads; this allows a node to upgrade and still have access to pre-existing that it might have had under primary index 1
  • After sufficient time has passed, a subsequent PR could be introduced to remove reading from primary index 1 as well. This will reduce IOPS for queries that aren't in the DB (no second read for primary index 1) as well as lowering overhead on a few other queries (ie getsSgnaturesForAddress).

Another minor win is in Blockstore::purge_columns_exact(); this function currently gets called on cluster restarts and takes quite a while to complete. Having a single primary index will cut the number of deletes in the write batch in half for nodes running with --enable-rpc-transaction-history (ie RPC nodes), and allow an early bail out for nodes without this flag (which should be the majority of staked validators).

Summary of Changes

@steviez steviez added the noCI Suppress CI on this Pull Request label Sep 22, 2023
@steviez
Copy link
Contributor Author

steviez commented Sep 22, 2023

@CriesofCarrots - I still need to fix up the unit tests, but wanted to get some thoughts on this one. The goal of this PR is to remove the dated primary index code while maintaining compatibility. Hypothetically, we could then simplify the reads on this in the future (ie only read primary index 0 instead of 0-then-1).

Another idea I just had while writing this up is that we could detect if there are any values with primary index 1, and if so, we can skip reading from that (knowing that we only write to primary index 1 with this change)

@steviez steviez force-pushed the bstore_phase_out_primary_index branch from 188fed3 to 5b1e4ec Compare October 5, 2023 15:00
@steviez
Copy link
Contributor Author

steviez commented Oct 10, 2023

Closing this PR; lots of the improvements this PR made have been shipped separately and finally ripping out the dead code will be performed in #33631

@steviez steviez closed this Oct 10, 2023
@steviez steviez deleted the bstore_phase_out_primary_index branch October 10, 2023 17:57
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
noCI Suppress CI on this Pull Request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant