You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Jan 22, 2025. It is now read-only.
Blockstore columns are cleaned in several ways, depending on the nature of the key
a) Columns that have a Slot as either the entire key or as the first element of the key can be range deleted by LedgerCleanupService. Items are deleted (ie marked as dead in the blockstore), and then a subsequent call to delete files that contain only keys in the cleaned range reclaims the storage in an efficient manner
b) Columns that have a Slot as part of the key but not the first element cannot be cleaned as efficiently. Rather, an atomic value is updated to indicate what slots have been cleaned by LedgerCleanupService. Background processes in rocksdb periodically iterate through all files, and compare the slot in the key to the cleaned range, deciding to keep or delete items on a per-key basis.
c) Several columns do not contain a slot in the key and as such, have no way of automatic cleaning. These columns are either deprecated or rely on explicit cleaning.
The TransactionMemos column falls in category c); however, there is currently no logic to clean this up. The key is a signature, which is not explicitly tied to a given slot. The slot could be indirectly determined by referencing another column that contains both signature and slot as part of the key, such as TransactionStatus.
As a result of the above, TransactionMemos is never cleaned and could technically grow without bound. Luckily, this column occupies much less space than the dominant columns, so this bloat may not be immediately apparent. Additionally, a node must be running with --enable-rpc-transaction-history to be storing this column
Proposed Solution
Come up with a scheme to efficiently clean this column. One such idea would be adjusting the key or value for TransactionMemos to contain the slot. This would allow the mechanism described in b) to work for this column family
The text was updated successfully, but these errors were encountered:
Problem
Blockstore columns are cleaned in several ways, depending on the nature of the key
a) Columns that have a
Slot
as either the entire key or as the first element of the key can be range deleted byLedgerCleanupService
. Items are deleted (ie marked as dead in the blockstore), and then a subsequent call to delete files that contain only keys in the cleaned range reclaims the storage in an efficient mannerb) Columns that have a
Slot
as part of the key but not the first element cannot be cleaned as efficiently. Rather, an atomic value is updated to indicate what slots have been cleaned byLedgerCleanupService
. Background processes in rocksdb periodically iterate through all files, and compare the slot in the key to the cleaned range, deciding to keep or delete items on a per-key basis.c) Several columns do not contain a slot in the key and as such, have no way of automatic cleaning. These columns are either deprecated or rely on explicit cleaning.
The
TransactionMemos
column falls in categoryc)
; however, there is currently no logic to clean this up. The key is a signature, which is not explicitly tied to a given slot. The slot could be indirectly determined by referencing another column that contains both signature and slot as part of the key, such asTransactionStatus
.As a result of the above,
TransactionMemos
is never cleaned and could technically grow without bound. Luckily, this column occupies much less space than the dominant columns, so this bloat may not be immediately apparent. Additionally, a node must be running with--enable-rpc-transaction-history
to be storing this columnProposed Solution
Come up with a scheme to efficiently clean this column. One such idea would be adjusting the key or value for
TransactionMemos
to contain the slot. This would allow the mechanism described in b) to work for this column familyThe text was updated successfully, but these errors were encountered: