feat(bitcoin)!: use mempool.space REST API

[ahonn]

Changes

Use mempool.space API instead of bitcoind and electrs

• APIs

  • For bitcoind part
    • ⚠️ getBlockchainInfo: we can assemble blockchain info by getting the tip block data (but the headers field cannot be obtained)
      • But fortunately, we hardly use this field before
    • sendRawTransaction: Exactly the same as postTx provided by mempool.space
  • For electrs part:
    • Most of the data is consistent since mempool.space is also based on electrs.
    • Objects use zod schema's parse to keep the interface consistent.

• Using Electrs as a fallback for mempool.space API

  • When mempool.space API is not available, electrs will be used as a fallback
  • For compatibility and scalability, electrs can be used as the primary data provider (see BITCOIN_DATA_PROVIDER)

• Error handles

  • No longer throw error codes related to Bitcoin RPC. (see src/services/bitcoind.ts)
  • Wrap BitcoinClientAPIError to handle mempool.space and electrs error
  • For BitcoinClientAPIError with a 404 status code, transaction processing needs to be delayed

Enhance Bitcoin transactions query

• implement paginated query of transactions
  • https://mempool.space/docs/api/rest#get-address-transactions
  • add after_txid param to get more transactions (/bitcoin/v1/address/:address/txs?after_txid=xxx )

Wrap recommended fee rates API

• details: Wrap the mempool recommended fee rates API #102
  • /bitcoin/v1/fees/recommended

Healthcheck updates

Remove bitcoind healthcheck and add mempool.space API healthcheck

Env variables

Added

/**
 * Bitcoin Mempool.space API URL
 * used to get bitcoin data and broadcast transaction
 */
BITCOIN_MEMPOOL_SPACE_API_URL: z.string(),
/**
 * Bitcoin data provider, support mempool and electrs
 * use mempool.space as default, electrs as fallback
 * change to electrs if you want to use electrs as default and mempool.space as fallback
 */
BITCOIN_DATA_PROVIDER: z.enum(['mempool', 'electrs']).default('mempool'),

BITCOIN_DATA_PROVIDER is used to determine which service to use as the primary data provider.

• When BITCOIN_DATA_PROVIDER=mempool
  • mempool.space API as primary data provider
  • electrs API as fallback data provider (BITCOIN_ELECTRS_API_URL is optional)
• When BITCOIN_DATA_PROVIDER=electrs
  • electrs API as primary data provider
  • mempool.space as fallback data provider (BITCOIN_MEMPOOL_SPACE_API_URL is optional)

Removed

We no longer need bitcoin RPC, we send transactions through mempool.space or our node indirectly through electrs

/**
 * The URL/USERNAME/PASSWORD of the Bitcoin JSON-RPC server.
 * The JSON-RPC server is used to query the Bitcoin blockchain.
 */
BITCOIN_JSON_RPC_URL: z.string(),
BITCOIN_JSON_RPC_USERNAME: z.string(),
BITCOIN_JSON_RPC_PASSWORD: z.string(),

Known Issues

• For issue electrs: return too many history entries when address has to many utxo/transactions record #89, mempool.space provides paging for txs, but no paging for address utxo. Therefore, more than 500 UTXOs will still encounter the problem of too many utxo.
  • cell.mempool.space is unlimited size for now (9000 size limit, almost unlimited)

Reviews

@Flouse @duanyytop @ShookLyngs @Dawn-githup @yuche

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
btc-assets-api	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 29, 2024 6:55am

[ahonn]

It was found that the version of mempool/mempool.js npm package is stuck at v2.3.0 released two years ago and does not support the after_txid parameter.
So I fork the repository to publish @cell-studio/mempool.js and use it instead of mempool/mempool.js .

Raised the relevant issue to them: mempool/mempool.js#191, and will replace it with the npm package officially provided by mempool after a new version is released in the future. (If mempool is not updated, then we can also consider maintaining it ourselves)

cc @Flouse @ShookLyngs