ERRO cannot compute the time passed

[Rugbertl]

Describe the bug
After several hours of running smoothly. Node just stopped synching. See logs below.

Mandatory Information

1. jcli 0.8.3 (HEAD-8f276c0, release, linux [x86_64]) - [rustc 1.39.0 (4560ea788 2019-11-04)]
2. jormungandr 0.8.3 (HEAD-8f276c0, release, linux [x86_64]) - [rustc 1.39.0 (4560ea788 2019-11-04)]

To Reproduce
Steps to reproduce the behavior:

1. Start stake pool
2. Wait x hours
3. Check state of node

Expected behavior
Node keeps synching.

Additional context
Dec 19 17:30:19.883 ERRO cannot compute the time passed since the last block: second time provided was later than self, task: stuck_notifier
Dec 19 17:34:20.712 WARN response canceled by the processing task, request: PushHeaders, sub_task: server, task: network
Dec 19 17:52:13.445 WARN response canceled by the processing task, request: PushHeaders, sub_task: server, task: network
Dec 19 17:52:13.445 WARN response canceled by the processing task, request: PushHeaders, sub_task: server, task: network
Dec 19 17:57:19.883 ERRO cannot compute the time passed since the last block: second time provided was later than self, task: stuck_notifier
Dec 19 17:58:19.883 ERRO cannot compute the time passed since the last block: second time provided was later than self, task: stuck_notifier

[Rugbertl]

Just found on v0.8.4

Dec 19 21:44:13.494 WARN response canceled by the processing task, request: PushHeaders, sub_task: server, task: network
Dec 19 21:48:15.166 ERRO cannot compute the time passed since the last block: second time provided was later than self, task: stuck_notifier
Dec 19 21:49:15.165 ERRO cannot compute the time passed since the last block: second time provided was later than self, task: stuck_notifier
Dec 19 21:50:15.165 ERRO cannot compute the time passed since the last block: second time provided was later than self, task: stuck_notifier

[rinor]

ERRO cannot compute the time passed since the last block: second time provided was later than self, task: stuck_notifier

This one is funny and weird. As far as I can tell the reason for this notification is related to some blocks produced from some pools that have the date in the future (according to your node). I have identifies at least 1 pool that produces blocks with date 2-3 minutes in the future. Probably that pool has the time wrong (on purpose or not). When that block hits your node and becomes your tip, the stuck_notifier tries to check the tome diff between your node local time and that block time, but since that block has a date in the future you see the message that second time provided, which is the time if the block, was later than self, which is the time of your node.

Now the story does not end here. There may be at least one other side effect caused by this kind of blocks, that affects the scheduled leader events that may cause that leader to not produce a block and be one of the cause for a message in your leader logs like:

reason: Not computing this schedule because of invalid state against the network blockchain

Please note that there are also legitimate cases that cause that message.

Update 2020-01-06: Not computing this schedule because of invalid state against the network blockchain should be already fixed in v0.8.5

[keynobase]

I also has this error :

Dec 29 20:43:09.571 WARN blockchain is not moving up, the last block was 122660 seconds ago, task: stuck_notifier

But the big problem is that my node stops receiving more hashses and I has to restart it.

version: jormungandr 0.8.5-3db0680

[rinor]

@Rugbertl is this still the case for v0.8.5? There are some changes made that should prevent this for happening frequently. Thank you

[rinor]

Please check also #1379 (comment). Will close this one for now, but please feel free to reopen/comment in case you encounter it again. Thank you.