feat(rpc): Implement an RPC transaction queue

[oxarbitrage]

Motivation

When send_raw_transaction rpc method is called, transactions can fail to get into the mempool. We need a queue of transactions created by the rpc call that can resubmit this failed transactions at a later time (after a block gets added).

Close #3654 if merged.

Solution

An rpc queue was implemented where retries are done.

Review

I think the code is pretty much complete now and have some decent test coverage. @jvff or @teor2345 will be good candidates for the review.

Reviewer Checklist

• Code implements Specs and Designs
• Tests for Expected Behaviour
• Tests for Errors

[teor2345]

This looks like a good design, I like the way you have separated the code out into different structs with clear roles. And it achieves the goals of the retry queue ticket.

Here are some alternative ways to achieve the same goals. And some things we need to be careful about. Let me know what you think!

Performance

I think we'll get better performance without the Mutex.

Here is how we could get the transactions instead:

• RpcImpl::send_raw_transaction() ->
• RpcImpl.transaction_sender ->
• channel ->
• Runner::run receiver

To make that happen, we could do something like:

1. Move the channel sender to RpcImpl
2. Move the channel receiver to Runner, and listen at the start of each run loop
3. Remove the Listener, its task, and the Mutex

Denial of Service

We're getting transactions from users all over the internet. Some of them might send lots of transactions, so we need to handle that well.

Currently, there is no limit on the size of the Queue.

And if the mpsc channel is full, it will block the send_raw_transaction RPC response until the next Runner loop. (The blocking happens after the changes above. Without them, the mutex might cause deadlocks.)

It's ok to drop extra transactions, because we already sent them to the mempool once. (I like the way this PR does that separately.)

Here are some ways to drop extra transactions:

1. Use a broadcast channel to automatically drop the oldest transactions when the channel is full
2. Add transactions to the front of the Queue, and remove them from the back if the queue gets full. You can use an IndexMap as an ordered HashMap.

This is similar to what the mempool does with its downloads and internal storage.

[teor2345]

@oxarbitrage I can do a review if you'd like, did you want to do it on a video call?

I think that will make it easier for me to understand some of the details.

[teor2345]

In PR #4046 I changed the return type of a method, and fixed up all the call sites.

But I think it's now used in more tests on the main branch.

I'll push a fix today.

[teor2345]

It was a trivial fix in a new test on the main branch, so I'm going to re-approve the PR with my fix.

[teor2345]

This PR failed with a temporary authorisation error:

#28 [auth] zealous-zebra/zebra/zebrad-test:pull,push token for us-docker.pkg.dev
[2031](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2031)
#28 DONE 0.0s
[2032](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2032)

[2033](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2033)
#27 exporting to image
[2034](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2034)
#27 pushing layers 0.6s done
[2035](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2035)
#27 ERROR: failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized
[2036](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2036)
------
[2037](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2037)
 > importing cache manifest from us-docker.pkg.dev/zealous-zebra/zebra/zebrad-test:4058-merge-buildcache:
[2038](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2038)
------
[2039](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2039)
------
[2040](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2040)
 > exporting to image:
[2041](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2041)
------
[2042](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2042)
error: failed to solve: failed to fetch oauth token: unexpected status: 401 Unauthorized
[2043](https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2043)
Error: buildx failed with: error: failed to solve: failed to fetch oauth token: unexpected status: 401 Unauthorized

https://github.com/ZcashFoundation/zebra/runs/5865028642?check_suite_focus=true#step:9:2043

[teor2345]

@Mergifyio refresh

[mergify]

refresh

✅ Pull request refreshed

[teor2345]

The merge queue's sync past mandatory checkpoint step failed with:

Error: No such container: klt-sync-checkpoint-4076-merge-da7aefa-yhww
https://github.com/ZcashFoundation/zebra/runs/5963709536?check_suite_focus=true#step:9:115

[teor2345]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated

[gustavovalverde]

@Mergifyio requeue

[mergify]

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically