fix: reduce redis data size, exclude UTXOs data from job return value

[ahonn]

We have observed that the bull:utxo-syncer-queue:events key data size in Redis is quite large due to the saving of UTXOs in the sync job return value.

This PR is aimed at reducing the Redis data size by removing the job return value from the UTXO sync and RGB++ collect jobs. Additionally, we are introducing some queue options when enqueuing a job to ensure that the job will be removed after it completes or fails.

We can safely remove the UTXOs data from the job return value because the UTXOs data cache is saved in the DataCache.

Some Facts

• By default, the BullMQ queue stream has a maximum approximated length of 10000 (https://api.docs.bullmq.io/interfaces/v5.QueueOptions.html#streams)
• Based on testing, the size of an event is approximately 4 KB. So the size of 10,000 events is about 40 MB.
• Partners have reported that the data size of bull:utxo-syncer-queue:events in Redis has reached as high as 1.4 GB. (The UTXOs stored in the job's return value cause this behavior.)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
btc-assets-api	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 11, 2024 6:12am