VC: Publish to multiple BNs

[lfinbob]

I have my lighthouse VC connected to 2 BNs (primary and fallback).
I propose to add a flag that makes the VC publish messages via all connected BNs concurrently.
That would sidestep bad fall-back experiences (see comment in 3002) with missed attestions/proposals/sync committees.
It would also increase the chances of messages being successfully published despite spotty internet connections with occasional packet loss on the beacon nodes.

Implementing this would make #3001 and #3002 not required anymore for solid fall-back.

[lfinbob]

Here's an additional use-case for this:

My lighthouse VC is connected to 2 different BNs. I then switched the first BN's EC from Besu to Geth, requiring a Geth sync. Which takes many hours.
For all that time the VC dumbly uses the degraded BN instead of switching to the healthy fallback BN.

The VC continues to log that it is connected to 2 BNs:

INFO Connected to beacon node(s) synced: 2, available: 2, total: 2, service: notifier

Meanwhile the VC still chooses the unhealthy BN to publish attestations. The unhealthy BN logs:

WARN Execution endpoint is not synced
ERRO Error updating deposit contract cache
WARN Head is optimistic [...] chain not fully verified, block and attestation production disabled until execution engine syncs
Attestation included in aggregate validator: [...]
INFO Attestation included in block validator: [...]

This makes fall back system pretty useless if one has to resync the EC.

[michaelsproul]

@lfinbob We're aware of the issues with the current fallback implementation and have opened a tracking issue to gather feedback, and refine the design: #3613. I think the behaviour you observed is a textbook instance of the problem described in that issue, and that the flood-publishing of this issue could be implemented in addition to improvements to the fallback system.

[lfinbob]

Great to read you guys are working on this, much appreciated.
I think flood publishing might be the fast & easy improvement, maybe a low-hanging fruit even.
Also it should have the secondary benefit of getting marginally better inclusion distances.

I vote for implementing this quickly, giving more time and thought to the more thorough enhancements described in #3613.
That would make the common two-machine-setup (1 primary, 1 fallback) for solo stakers and Rocket Pool node operators work smoothly with lighthouse immediately.

[xrchz]

I had a block proposal fail to use the builder network on my fallback because my primary was still syncing - I suspect flood publishing would have avoided this. Very keen to see it soon.