reload of w3s

[franjoespejo]

Describe the bug
When the connection with the w3s is closed and the requests do timeout, if you enable the w3s, the connection become active again and the pubkeys reloaded start to detach and attach pubkeys from the w3s but after they do this the logs stop and validator freezes but doesnt crash

To Reproduce
Steps to reproduce the behavior:

1. Platform details (OS, architecture): 'amd64, ubuntu'
2. Branch/commit used: '
   version{version="v24.5.1-d2a075-stateofus",commit="d2a075"} 1.0
   version{version="v24.7.0-99f657-stateofus",commit="99f657"} 1.0
   version{version="v24.9.0-f54a03-stateofus",commit="f54a03"} 1.0
   '
3. Commands being executed: '
   ENTRYPOINT:
   nimbus_validator_client
   CMD:
   --doppelganger-detection=false
   --non-interactive
   --web3-signer-update-interval=10
   --beacon-node=http://172.16.0.13:4000
   --suggested-fee-recipient=0x7d58F316a09064cA16c23Ad9c8e3Ab37E51815Ff
   --graffiti=val1
   --metrics
   --metrics-address=0.0.0.0
   --metrics-port=34050
   --web3-signer-url=http://diva5:9000
   '
4. Relevant log lines:

'WRN 2024-10-06 16:20:31.003+00:00 Unable to sign committee message using remote signer service=sync_committee_service slot=1136 validator=b43ccb05@67 block_root=632972d4
WRN 2024-10-06 16:20:31.003+00:00 Unable to sign attestation                 reason="Operation timed out" service=attestation_service validator=b97dbe4a@72
WRN 2024-10-06 16:20:31.004+00:00 Unable to sign attestation                 reason="Operation timed out" service=attestation_service validator=b97ecbcf@65
WRN 2024-10-06 16:20:31.166+00:00 Unable to resolve validator's source distributed signer address reason="Unable to resolve remote hostname" web3signer_url=http://diva1:9000
INF 2024-10-06 16:20:32.001+00:00 Slot start                                 slot=1137 epoch=142 attestationIn=2s blockIn=12s validators=10 node_status=synced delay=1ms540us769ns
WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137
WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137
WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137
WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1138
WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1138
WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1138
WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1139
WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1139
WRN 2024-10-06 16:20:32.195+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1139
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1140
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1140
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1140
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1141
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1141
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1141
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1142
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1142
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1142
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1143
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1143
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1143
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1144
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1144
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1144
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1145
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1145
WRN 2024-10-06 16:20:32.196+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1145
WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137
WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137
WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1137
WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1139
WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1139
WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1141
WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1141
WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=142 slot=1142
WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1144
WRN 2024-10-06 16:20:32.197+00:00 Unable to create slot signature using remote signer reason="Communication failed while sending/receiving request, http error [HttpConnectionError]: Could not connect to remote host" epoch=143 slot=1144
WRN 2024-10-06 16:20:34.140+00:00 Unable to sign committee message using remote signer service=sync_committee_service slot=1137 validator=b43ccb05@67 block_root=d307bfd3
WRN 2024-10-06 16:20:34.140+00:00 Unable to sign attestation                 reason="Remote signer returns 500 Internal Server Error" service=attestation_service validator=ac30aacd@66
WRN 2024-10-06 16:20:34.140+00:00 Unable to sign committee message using remote signer service=sync_committee_service slot=1137 validator=8de7ec50@64 block_root=d307bfd3
WRN 2024-10-06 16:20:34.140+00:00 Unable to sign committee message using remote signer service=sync_committee_service slot=1137 validator=ac30aacd@66 block_root=d307bfd3
WRN 2024-10-06 16:20:34.142+00:00 Unable to sign attestation                 reason="Remote signer returns 500 Internal Server Error" service=attestation_service validator=a39731d5@71
WRN 2024-10-06 16:20:34.142+00:00 Unable to sign attestation                 reason="Remote signer returns 500 Internal Server Error" service=attestation_service validator=8de7ec50@64
NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached          topics="val_pool" pubkey=ac30aacd9e91cb0727c34ca6b40fbfd4d255b998471e25c443cf6cf777d6bb823a58e162958f32c3c5ca80453387a5d2 validator=ac30aacd
NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached          topics="val_pool" pubkey=966c488d807b3208bb1b10a1af422bac8d363c8015cda4e24d214549ced019cd3dd575545dd887461cae3f70d95cb061 validator=966c488d
NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached          topics="val_pool" pubkey=b8c6663371dc37bf083134dea26a20115ccc52b7c15a662bcfa33435e4ade14c6bc9714a5cdee492530accf8a327b2aa validator=b8c66633
NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached          topics="val_pool" pubkey=b3faeebfbebd085b9123ae0e09af9cd15d3b1db6a25f3e82d8b48b68e53522b41b342a3a3c8b008897df356048862d98 validator=b3faeebf
NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached          topics="val_pool" pubkey=b97dbe4add8aefd96c575ae9de19d1ac590bb7d92f23a9e4e113f7271c2243cf689e7645879efbb546d58ec44f5263d6 validator=b97dbe4a
NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached          topics="val_pool" pubkey=a39731d5cb52838d02d4ff897ab908c0f76a9ef837f9288c634ed3091a1f69d5347dc65cd2c8009a5207a369a4c6bdae validator=a39731d5
NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached          topics="val_pool" pubkey=8de7ec501d574152f52a962bf588573df2fc3563fd0c6077651208ed20f24f3d8572425706b343117b48bdca56808416 validator=8de7ec50
NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached          topics="val_pool" pubkey=b97ecbcfe8c52b9bcdca9e75da13c5650b751b037c570934ea6b6441ff32de6566c50dafc0557e63105b2ee7e8cbb39e validator=b97ecbcf
NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached          topics="val_pool" pubkey=95833097520df43a5cb013e97f80041a7a0b7d84a4ec79e2f16baeeb6edfbcf62ede97becfde73883831bb65e1415dc0 validator=95833097
NTC 2024-10-06 16:20:36.167+00:00 Dynamic remote validator detached          topics="val_pool" pubkey=b43ccb05317c2b666470ab251e987d6bf31f5ead6b5edac5fe007dd334ae6ce1a92e24c19e5ab387cd8fa253b63bb78c validator=b43ccb05
INF 2024-10-06 16:20:38.002+00:00 Slot start                                 slot=1138 epoch=142 attestationIn=8s blockIn=<unknown> validators=0 node_status=synced delay=2ms728us102ns
INF 2024-10-06 16:20:44.001+00:00 Slot start                                 slot=1139 epoch=142 attestationIn=2s blockIn=<unknown> validators=0 node_status=synced delay=1ms402us782ns
NTC 2024-10-06 16:20:46.169+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=b8c6663371dc37bf083134dea26a20115ccc52b7c15a662bcfa33435e4ade14c6bc9714a5cdee492530accf8a327b2aa validator=b8c66633 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: b8c6663371dc37bf083134dea26a20115ccc52b7c15a662bcfa33435e4ade14c6bc9714a5cdee492530accf8a327b2aa)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000
NTC 2024-10-06 16:20:46.170+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=a765e534ca5f091ef4056dac0b84bcdbc90751bc0950b46cccebbe569f3f6e8afc964cf7d34adfed9b444e6d6d300d6d validator=a765e534 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: a765e534ca5f091ef4056dac0b84bcdbc90751bc0950b46cccebbe569f3f6e8afc964cf7d34adfed9b444e6d6d300d6d)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000
NTC 2024-10-06 16:20:46.170+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=b97ecbcfe8c52b9bcdca9e75da13c5650b751b037c570934ea6b6441ff32de6566c50dafc0557e63105b2ee7e8cbb39e validator=b97ecbcf remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: b97ecbcfe8c52b9bcdca9e75da13c5650b751b037c570934ea6b6441ff32de6566c50dafc0557e63105b2ee7e8cbb39e)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000
NTC 2024-10-06 16:20:46.170+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=b97dbe4add8aefd96c575ae9de19d1ac590bb7d92f23a9e4e113f7271c2243cf689e7645879efbb546d58ec44f5263d6 validator=b97dbe4a remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: b97dbe4add8aefd96c575ae9de19d1ac590bb7d92f23a9e4e113f7271c2243cf689e7645879efbb546d58ec44f5263d6)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000
NTC 2024-10-06 16:20:46.170+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=966c488d807b3208bb1b10a1af422bac8d363c8015cda4e24d214549ced019cd3dd575545dd887461cae3f70d95cb061 validator=966c488d remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: 966c488d807b3208bb1b10a1af422bac8d363c8015cda4e24d214549ced019cd3dd575545dd887461cae3f70d95cb061)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000
NTC 2024-10-06 16:20:46.170+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=b3faeebfbebd085b9123ae0e09af9cd15d3b1db6a25f3e82d8b48b68e53522b41b342a3a3c8b008897df356048862d98 validator=b3faeebf remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: b3faeebfbebd085b9123ae0e09af9cd15d3b1db6a25f3e82d8b48b68e53522b41b342a3a3c8b008897df356048862d98)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000
NTC 2024-10-06 16:20:46.171+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=8de7ec501d574152f52a962bf588573df2fc3563fd0c6077651208ed20f24f3d8572425706b343117b48bdca56808416 validator=8de7ec50 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: 8de7ec501d574152f52a962bf588573df2fc3563fd0c6077651208ed20f24f3d8572425706b343117b48bdca56808416)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000
NTC 2024-10-06 16:20:46.171+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=87df22a26f59f2854d697ecfda8903f3ea6dd9ec91ae7effb5ff6ab0d599b7d3c57c1137d21522e1ab2adb8887e3a1c4 validator=87df22a2 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: 87df22a26f59f2854d697ecfda8903f3ea6dd9ec91ae7effb5ff6ab0d599b7d3c57c1137d21522e1ab2adb8887e3a1c4)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000
NTC 2024-10-06 16:20:46.171+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=ac30aacd9e91cb0727c34ca6b40fbfd4d255b998471e25c443cf6cf777d6bb823a58e162958f32c3c5ca80453387a5d2 validator=ac30aacd remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: ac30aacd9e91cb0727c34ca6b40fbfd4d255b998471e25c443cf6cf777d6bb823a58e162958f32c3c5ca80453387a5d2)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000
NTC 2024-10-06 16:20:46.171+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=95833097520df43a5cb013e97f80041a7a0b7d84a4ec79e2f16baeeb6edfbcf62ede97becfde73883831bb65e1415dc0 validator=95833097 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: 95833097520df43a5cb013e97f80041a7a0b7d84a4ec79e2f16baeeb6edfbcf62ede97becfde73883831bb65e1415dc0)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000
NTC 2024-10-06 16:20:46.171+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=b43ccb05317c2b666470ab251e987d6bf31f5ead6b5edac5fe007dd334ae6ce1a92e24c19e5ab387cd8fa253b63bb78c validator=b43ccb05 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: b43ccb05317c2b666470ab251e987d6bf31f5ead6b5edac5fe007dd334ae6ce1a92e24c19e5ab387cd8fa253b63bb78c)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000
NTC 2024-10-06 16:20:46.172+00:00 Dynamic remote validator attached          topics="val_pool" pubkey=a39731d5cb52838d02d4ff897ab908c0f76a9ef837f9288c634ed3091a1f69d5347dc65cd2c8009a5207a369a4c6bdae validator=a39731d5 remote_signer="@[(url: http://diva1:9000, id: 0, pubkey: a39731d5cb52838d02d4ff897ab908c0f76a9ef837f9288c634ed3091a1f69d5347dc65cd2c8009a5207a369a4c6bdae)]" initial_fee_recipient=7d58f316a09064ca16c23ad9c8e3ab37e51815ff initial_gas_limit=30000000'

that's all the logs, the validator keeps running but doesnt work nor log

[franjoespejo]

it happens for both minimal and mainnet preset with different interval values from 10 to 360s

[cheatfate]

It looks like web3signer client unable to resolve hostname of web3signer host which is diva1.

[tersec]

#6615