Skip to content
This repository has been archived by the owner on Jan 22, 2025. It is now read-only.

TransactionMemos column has no mechanism to remove data #32519

Closed
steviez opened this issue Jul 18, 2023 · 0 comments · Fixed by #33419
Closed

TransactionMemos column has no mechanism to remove data #32519

steviez opened this issue Jul 18, 2023 · 0 comments · Fixed by #33419

Comments

@steviez
Copy link
Contributor

steviez commented Jul 18, 2023

Problem

Blockstore columns are cleaned in several ways, depending on the nature of the key
a) Columns that have a Slot as either the entire key or as the first element of the key can be range deleted by LedgerCleanupService. Items are deleted (ie marked as dead in the blockstore), and then a subsequent call to delete files that contain only keys in the cleaned range reclaims the storage in an efficient manner
b) Columns that have a Slot as part of the key but not the first element cannot be cleaned as efficiently. Rather, an atomic value is updated to indicate what slots have been cleaned by LedgerCleanupService. Background processes in rocksdb periodically iterate through all files, and compare the slot in the key to the cleaned range, deciding to keep or delete items on a per-key basis.
c) Several columns do not contain a slot in the key and as such, have no way of automatic cleaning. These columns are either deprecated or rely on explicit cleaning.

The TransactionMemos column falls in category c); however, there is currently no logic to clean this up. The key is a signature, which is not explicitly tied to a given slot. The slot could be indirectly determined by referencing another column that contains both signature and slot as part of the key, such as TransactionStatus.

As a result of the above, TransactionMemos is never cleaned and could technically grow without bound. Luckily, this column occupies much less space than the dominant columns, so this bloat may not be immediately apparent. Additionally, a node must be running with --enable-rpc-transaction-history to be storing this column

Proposed Solution

Come up with a scheme to efficiently clean this column. One such idea would be adjusting the key or value for TransactionMemos to contain the slot. This would allow the mechanism described in b) to work for this column family

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant