Fix frequent Zebra hangs during syncing

[teor2345]

Motivation

Zebra is currently hanging during about 5% - 90% of syncs, depending on the machine and network. But Zebra doesn't seem to hang after the initial sync.

This Epic contains bugs that might be causing these hangs.

But it's also possible that:

• bugs we're currently working on will fix the hangs, or
• other undiscovered bugs are causing the hangs.

So we don't want to add these bugs to #3096 yet.

Tasks

First, fix the bugs we are currently working on, and see if Zebra stops hanging all the time:

• Fix syncer download order and add sync tests #3168
• Add outbound peer message timeout #3234
• Await both the heartbeat request and shutdown channel in the heartbeat task #3254
• Deadlock when using LatestChainTip #3379

If Zebra keeps on hanging frequently, implement these diagnostics:

• Integrate tokio-console #2112

Then use the diagnostics to:

• prioritise the bugs in this epic, or
• open new bugs for any issues reported by tokio-console.

Likely Causes

This code has caused hangs before:

• Avoid deadlocks in the crawler by doing address book updates and handshakes in separate tasks #3288
• Security: Remove the ErrorSlot from the heartbeat and connection tasks #3263
• Make sync, inbound, and block verifier check if a block hash is in any chain or any queue #862
• Move finalized state database and filesystem operations to blocking threads to avoid hangs #2188
• Move the metrics recorder to a blocking thread to avoid hangs #2201
• Move batch cryptography flushes to a blocking thread to avoid hangs #2202
• Zebra validation can be very slow on macOS, and cause very slow syncs #3042
• Interrupt handler does not work when a blocking task is running #1351

[mpguerra]

Do we need to review this issue and its associated potential fixes given that it seems we've fixed some hangs for now...

[conradoplg]

I think we can close this after #3234 is completed

[teor2345]

I'm not seeing any hangs after testing the latest main on 4 different mainnet and testnet instances, so I'm happy to just close this now.