fix: Move access to database into a blocking task

[ameten]

Description

Move access to database into a blocking task

The access to rocksdb is synchronous and we discovered that it take about 1 millisecond to make one retrieval of an insertion into a merkle tree from the database when we build internal merkle tree in relayer from scratch. As the result, the worker thread which executes the task, is blocked by access to the database and cannot be used to execute other scheduled tasks.

This change moves the access to the database into a separate thread in the blocking pool. It frees up the worker thread to execute other tasks.

Related issues

• Contributes into https://github.com/hyperlane-xyz/issues/issues/1420

Backward compatibility

Yes

Testing

Manual run of relayer locally

[changeset-bot]

⚠️ No Changeset found

Latest commit: b2635e5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.53%. Comparing base (d6ddf5b) to head (b2635e5).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5612   +/-   ##
=======================================
  Coverage   77.53%   77.53%           
=======================================
  Files         103      103           
  Lines        2110     2110           
  Branches      190      190           
=======================================
  Hits         1636     1636           
  Misses        453      453           
  Partials       21       21           

Components	Coverage Δ
core	87.80% <ø> (ø)
hooks	79.39% <ø> (ø)
isms	83.68% <ø> (ø)
token	91.27% <ø> (ø)
middlewares	79.80% <ø> (ø)

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.