Requests stuck in pending state

[alexsporn]

The node got unsync because the processor got full (slowed down forcefully over INX to force it to fill up). After removing the slowdown the node started syncing again but got stuck a couple of milestones later due to 5 requests staying pending forever.

I could reproduce this in the private tangle twice and both times 5 requests were stuck pending.

[alexsporn]

The problem does not lie in the request queue, this is working correctly. It happens if you e.g. have only 1 peer and the connections gets stuck. If no more heartbeats are received, then no more requests will be sent out to this peer.
Adding a second peer while this state happens clears up the pending queue and the node syncs again.
The stuck peer only recovers if it is re-added (e.g. by editing it using the dashboard)

[alexsporn]

The pending queue cleared up, but it seems there are still blocks missing which are not added to the queue, although the queue is empty:

2022-09-14T09:17:15+02:00       INFO    Tangle  Valid milestone detected! Index: 477
2022-09-14T09:17:15+02:00       INFO    Tangle  Run solidity check for Milestone (429) ...
2022-09-14T09:17:15+02:00       WARN    Tangle  Stopped solidifier due to missing block -> Requested missing blocks (0/6), collect: 0s
2022-09-14T09:17:15+02:00       INFO    WarpSync        Requesting missing milestones 429 - 579
2022-09-14T09:17:15+02:00       INFO    Tangle  Milestone couldn't be solidified! 429
2022-09-14T09:17:15+02:00       INFO    WarpSync        Target updated to milestone 477 (checkpoint 191)
req(qu/pe/proc/lat): 00000/00000/00000/0000ms, reqQMs: 0, processor: 00000, CMI/LMI: 191/477, BPS (in/new/out): 00153/00001/00000, Tips (non-/semi-lazy): 56/0
req(qu/pe/proc/lat): 00000/00000/00000/0000ms, reqQMs: 0, processor: 00000, CMI/LMI: 191/477, BPS (in/new/out): 00153/00001/00000, Tips (non-/semi-lazy): 56/0
req(qu/pe/proc/lat): 00000/00000/00000/0000ms, reqQMs: 0, processor: 00000, CMI/LMI: 191/477, BPS (in/new/out): 00150/00000/00000, Tips (non-/semi-lazy): 56/0

[alexsporn]

The solidifier is triggering for milestone 429 instead of 192.
The node is missing milestone 192-428 it seems

The 6 blocks that is trying to request are the parents of milestone 429