Skip to content

Commit

Permalink
Reduce some EE and builder related ERRO logs to WARN (sigp#3966)
Browse files Browse the repository at this point in the history
NA

Our `ERRO` stream has been rather noisy since the merge due to some unexpected behaviours of builders and EEs. Now that we've been running post-merge for a while, I think we can drop some of these `ERRO` to `WARN` so we're not "crying wolf".

The modified logs are:

I'm seeing this quite frequently on Geth nodes. They seem to timeout when they're busy and it rarely indicates a serious issue. We also have logging across block import, fork choice updating and payload production that raise `ERRO` or `CRIT` when the EE times out, so I think we're not at risk of silencing actual issues.

In sigp#3775 we reduced this log from `CRIT` to `ERRO` since it's common for builders to fail to reveal the block to the producer directly whilst still broadcasting it to the networ. I think it's worth dropping this to `WARN` since it's rarely interesting.

I elected to stay with `WARN` since I really do wish builders would fulfill their API promises by returning the block to us. Perhaps I'm just being pedantic here, I could be convinced otherwise.

It seems like builders and/or mev-boost struggle to handle heavy loads of validator registrations. I haven't observed issues with validators not actually being registered, but I see timeouts on these endpoints many times a day. It doesn't seem like this `ERRO` is worth it.

This means we failed to respond to a peer on the P2P network with a block they requested because of an error in the `execution_layer`. It's very common to see timeouts or incomplete responses on this endpoint whilst the EE is busy and I don't think it's important enough for an `ERRO`. As long as the peer count stays high, I don't think the user needs to be actively concerned about how we're responding to peers.

NA
  • Loading branch information
paulhauner authored and Woodpile37 committed Jan 6, 2024
1 parent aa33171 commit da80323
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
2 changes: 1 addition & 1 deletion beacon_node/execution_layer/src/engines.rs
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,7 @@ impl Engine {
Ok(result)
}
Err(error) => {
error!(
warn!(
self.log,
"Execution engine call failed";
"error" => ?error,
Expand Down
2 changes: 1 addition & 1 deletion beacon_node/http_api/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2903,7 +2903,7 @@ pub fn serve<T: BeaconChainTypes>(
.await
.map(|resp| warp::reply::json(&resp))
.map_err(|e| {
error!(
warn!(
log,
"Relay error when registering validator(s)";
"num_registrations" => filtered_registration_data.len(),
Expand Down

0 comments on commit da80323

Please sign in to comment.