Blockstore special columns: minimize deletes in PurgeType::Exact

[CriesofCarrots]

Problem

Currently, Blockstore::purge_columns_exact() gets called on cluster restarts and takes quite a while to complete. However, we can halve the number of special-column deletes in the write batch in half for nodes running with --enable-rpc-transaction-history (ie RPC nodes), and eliminate them altogether for nodes without this flag (the majority of staked validators).

Summary of Changes

We already read both values from the transaction_status_index_cf, so use this data in memory to determine which (if any) primary_index contains the slot in question.

This is a reworking of functionality that is in #33370, but done in a way that works better with #33419

[CriesofCarrots]

We might still want the special-column-empty check from #33370, however, once I muck this up by removing the primary indexes.

[codecov]

Codecov Report

Merging #33498 (317048c) into master (9761e6f) will increase coverage by 0.0%.
Report is 1 commits behind head on master.
The diff coverage is 97.7%.

@@           Coverage Diff           @@
##           master   #33498   +/-   ##
=======================================
  Coverage    81.7%    81.7%           
=======================================
  Files         802      802           
  Lines      217832   217870   +38     
=======================================
+ Hits       178117   178153   +36     
- Misses      39715    39717    +2     

[steviez]

We might still want the special-column-empty check from #33370,

Will give proper review tomorrow, but 100% agreed that checking for empty columns is a worthwhile change. I can break that out from 33370 as well; it would be logic in the purge exact function + logic to remove the dummy entries at primary index 2 (those entries are completely unused nowadays so we can safely remove while being backwards compatible).

[steviez]

LGTM and 🚀 for adding the unit test coverage too!

[steviez]

One of the original motivations for #33370 was that I had observed committing the massive write batch took much longer in comparison to iterating the blocks and finding tx sigs / address pubkeys to delete. I had mentioned this in DM to Tyera too, but I hadn't posted actual numbers so here are some to provide a public paper trail.

I hacked up validator to perform backup_and_clear_blockstore on 9k slots (1 hour of blocks @ 400ms) on a ledger that had reached the default (200M) --limit-ledger-size on mnb. Here is the datapoint with relevant breakdowns with the current tip of master (does NOT include this PR or mine to reduce number of ops).

[2023-10-03T20:43:25.305112736Z INFO  solana_metrics::metrics] datapoint: blockstore-purge
from_slot=221364739i to_slot=221373739i
delete_range_us=           83_441_130i
write_batch_us=           648_443_741i
delete_files_in_range_us= 648_443_741i

Some thoughts / summary:

• delete_range_us is almost exclusively spent iterating sand deserializing the blocks in Blockstore::purge_special_columns_exact() and took 83 seconds
  • Something on the order of 10's of micros is spent calling delete_range_cf() for all of the slot based columns total
• Committing the write batch took 648 seconds
• So, nearly an 8:1 ratio of time spend committing write batch vs. iterating blocks
• With this PR, write batch size should be cut in half, so I'd also expect commit time to drop in half
• Hypothetically, I'd expect blocks around a cluster outage to be less full so the ratios may not be as pronounced there, but regardless, the change this PR makes is worthwhile
• The delete_files_in_range_us number being identical to write_batch_us revealed a bug to me (Fix blockstore-purge delete_files_in_range_us metric #33512) but that method read minimal metadata for each SST, so I suspect it should be pretty quick

Lastly, the numbers being so large provide justification on why having a check for empty columns is of value; I still plan on pushing that separately

[CriesofCarrots]

Sorry @steviez , can a get a re-✅ ?
I forgot to update my compaction-filter test for the extra transaction.