Move libp2p to worker thread

[dapplion]

Rationale

Main goals:

• Move network crypto to thread
• Isolate network to protect main thread of DOS attacks
• Allow to throttle network processor better to keep main thread event loop lag under control

Also, for nodes subscribed to many subnets recent benchmarks show that network handling takes decent chunk CPU time, lagging the main thread and delaying other pipelines like block processing.

• Part of Move libp2p,discv5,backfill to workers #4856
• WIP branch all-in-one feat: move libp2p to network worker thread #5229

Architecture

Simple and lame chart for general idea. libp2p instance is rooted on the TCP transport's socket. It pulls many components that are best together in the same thread:

• peer manager
• gossip
• reqresp

On the other hand, the main thread is the only thread with access to the state cache, so the network processor must remain there. For now we'll place the network queues in the main thread, but those could be placed in the worker.

Interfaces

To do A/B testing we want to have a flag to switch the libp2p instance from being in a worker thread or the main thread. So we need

• A) Main thread "public" facing network API with variable backend. Implements NetworkPublic and calls NetworkCore internally (either B or C).
• B) Internal network class with logic de-duplicated from either backend. Implements NetworkCore with actual logic
• C) Main to worker event based interface to wire A to B through worker. Implements NetworkCore and calls NetworkCore internally (B).

interface NetworkPublic extends NetworkCorePublic {
  // Functions using multiple reqresp / gossip methods
  publishBeaconBlockMaybeBlobs(signedBlock: BlockInput): Promise<void>;
  beaconBlocksMaybeBlobsByRange(peerId: PeerId, request: phase0.BeaconBlocksByRangeRequest): Promise<BlockInput[]>;
  beaconBlocksMaybeBlobsByRoot(peerId: PeerId, request: phase0.BeaconBlocksByRootRequest): Promise<BlockInput[]>;
  
  // ReqResp caller helpers
  // NOTE: Should map to a single fn to prevent boilerplate
  status(peerId: PeerId, request: phase0.Status): Promise<phase0.Status>;
  goodbye(peerId: PeerId, request: phase0.Goodbye): Promise<void>;
  ping(peerId: PeerId): Promise<phase0.Ping>;
  metadata(peerId: PeerId): Promise<allForks.Metadata>;
  beaconBlocksByRange(peerId: PeerId, request: phase0.BeaconBlocksByRangeRequest): Promise<allForks.SignedBeaconBlock[]>;
  beaconBlocksByRoot(peerId: PeerId, request: phase0.BeaconBlocksByRootRequest): Promise<allForks.SignedBeaconBlock[]>;
  blobsSidecarsByRange(peerId: PeerId, request: deneb.BlobsSidecarsByRangeRequest): Promise<deneb.BlobsSidecar[]>;
  beaconBlockAndBlobsSidecarByRoot(peerId: PeerId, request: deneb.BeaconBlockAndBlobsSidecarByRootRequest): Promise<deneb.SignedBeaconBlockAndBlobsSidecar[]>;
  lightClientBootstrap(peerId: PeerId, request: Uint8Array): Promise<allForks.LightClientBootstrap>;
  lightClientOptimisticUpdate(peerId: PeerId): Promise<allForks.LightClientOptimisticUpdate>;
  lightClientFinalityUpdate(peerId: PeerId): Promise<allForks.LightClientFinalityUpdate>;
  lightClientUpdatesByRange(peerId: PeerId, request: altair.LightClientUpdatesByRange): Promise<allForks.LightClientUpdate[]>;

  // Gossip publish helpers
  // NOTE: This functions could be stand-alone since each have usually one single caller
  publishBeaconBlock(signedBlock: allForks.SignedBeaconBlock): Promise<PublishResult>
  publishSignedBeaconBlockAndBlobsSidecar(item: deneb.SignedBeaconBlockAndBlobsSidecar): Promise<PublishResult>
  publishBeaconAggregateAndProof(aggregateAndProof: phase0.SignedAggregateAndProof): Promise<PublishResult>
  publishBeaconAttestation(attestation: phase0.Attestation, subnet: number): Promise<PublishResult>
  publishVoluntaryExit(voluntaryExit: phase0.SignedVoluntaryExit): Promise<PublishResult>
  publishBlsToExecutionChange(blsToExecutionChange: capella.SignedBLSToExecutionChange): Promise<PublishResult>
  publishProposerSlashing(proposerSlashing: phase0.ProposerSlashing): Promise<PublishResult>
  publishAttesterSlashing(attesterSlashing: phase0.AttesterSlashing): Promise<PublishResult>
  publishSyncCommitteeSignature(signature: altair.SyncCommitteeMessage, subnet: number): Promise<PublishResult>
  publishContributionAndProof(contributionAndProof: altair.SignedContributionAndProof): Promise<PublishResult>
  publishLightClientFinalityUpdate(lightClientFinalityUpdate: allForks.LightClientFinalityUpdate): Promise<PublishResult>
  publishLightClientOptimisticUpdate(lightClientOptimisitcUpdate: allForks.LightClientOptimisticUpdate): Promise<PublishResult>
}

interface NetworkCore extends NetworkCorePublic {
  // Service
  metrics(): Promise<string>;
  close(): Promise<void>;

  sendReqRespRequest(method, peerId: PeerId, request): ??;
  publishGossip<K extends GossipType>(topic: GossipTopicMap[K], object: GossipTypeMap[K], opts?: PublishOpts): Promise<number>;
}

interface NetworkCorePublic {
  getEnr(): Promise<SignableENR | undefined>;
  getMetadata(): Promise<altair.Metadata>;
  getConnectedPeers(): PeerId[];
  getConnectedPeerCount(): number;

  prepareBeaconCommitteeSubnet(subscriptions: CommitteeSubscription[]): Promise<void>;
  prepareSyncCommitteeSubnets(subscriptions: CommitteeSubscription[]): Promise<void>;
  reStatusPeers(peers: PeerId[]): Promise<void>;
  reportPeer(peer: PeerId, action: PeerAction, actionName: string): Promise<void>;

  subscribeGossipCoreTopics(): Promise<void>;
  unsubscribeGossipCoreTopics(): Promise<void>;
  isSubscribedToGossipCoreTopics(): boolean;

  connectToPeer(peer: PeerId, multiaddr: Multiaddr[]): Promise<void>;
  disconnectPeer(peer: PeerId): Promise<void>;
  dumpPeers(): Promise<routes.lodestar.LodestarNodePeer[]>;
  dumpPeer(peerIdStr: string): Promise<routes.lodestar.LodestarNodePeer | undefined>;
  dumpPeerScoreStats(): Promise<PeerScoreStats>;
  dumpGossipPeerScoreStats(): Promise<PeerScoreStatsDump>;
  dumpGossipQueue(gossipType: GossipType): Promise<PendingGossipsubMessage[]>;
  dumpDiscv5KadValues(): Promise<string[]>;
}

[philknows]

Is this technically complete now with #5229 ?