Frequent reorgs on parachain

[Mihajlo-Pavlovic]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

Since enabling Asynchronous Backing with runtime upgrade and updating dependencies to v1.11.0, we started experiencing frequent reorgs on the Neuroweb parachain.
Collators produce a block, right after that, they drop it and collate a new one on its spot.
It seems reorgs coincide with forks on the relay chain.

Here is an example of collator logs when reorg happens:

2025-01-14 10:28:00 [Relaychain] 🏆 Imported #24282579 (0xd509…4f8a → 0xdbcc…4d3c)
2025-01-14 10:28:00 [Parachain] 🏆 Imported #7514441 (0x4d7a…3c85 → 0xc4da…85b8)
2025-01-14 10:28:00 [Parachain] 🆕 Imported #7514441 (0x4d7a…3c85 → 0x07cf…31e9)
2025-01-14 10:28:00 [Relaychain] ♻️  Reorg on #24282579,0xdbcc…4d3c to #24282579,0x91ee…4d6c, common ancestor #24282578,0xd509…4f8a
2025-01-14 10:28:00 [Relaychain] 🏆 Imported #24282579 (0xd509…4f8a → 0x91ee…4d6c)
2025-01-14 10:28:01 [Relaychain] 💤 Idle (8 peers), best: #24282579 (0x91ee…4d6c), finalized #24282576 (0x6d79…995e), ⬇ 199.9kiB/s ⬆ 532.7kiB/s
2025-01-14 10:28:01 [Parachain] 💤 Idle (15 peers), best: #7514441 (0xc4da…85b8), finalized #7514436 (0x449b…d2fa), ⬇ 12.0kiB/s ⬆ 11.1kiB/s
2025-01-14 10:28:06 [Relaychain] 🏆 Imported #24282580 (0x91ee…4d6c → 0xeada…2afe)
2025-01-14 10:28:06 [Parachain] 🙌 Starting consensus session on top of parent 0x07cf9cc6a987f9854823ce2bcc7dfc0435614f158f78aca00a3e5724295331e9
2025-01-14 10:28:06 [Parachain] 🎁 Prepared block for proposing at 7514442 (7 ms) [hash: 0x23439ddfdd9b47ce2bef5a1e7f4f3bd8567bcbe075a4e5ff094a3d0a6199e300; parent_hash: 0x07cf…31e9; extrinsics (7): [0xd105…a3ce, 0x7317…2651, 0x118f…3b62, 0x5f26…55ab, 0xaa4f…d3db, 0x94bb…9d55, 0x852f…d66d]
2025-01-14 10:28:06 [Parachain] 🔖 Pre-sealed block for proposal at 7514442. Hash now 0xe038ac630adefc1d5a0952566a16b6a851a2ffc2da17ce6e066fd347bfaee0d7, previously 0x23439ddfdd9b47ce2bef5a1e7f4f3bd8567bcbe075a4e5ff094a3d0a6199e300.
2025-01-14 10:28:06 [Parachain] ♻️  Reorg on #7514441,0xc4da…85b8 to #7514442,0xe038…e0d7, common ancestor #7514440,0x4d7a…3c85
2025-01-14 10:28:06 [Parachain] 🏆 Imported #7514442 (0x07cf…31e9 → 0xe038…e0d7)
2025-01-14 10:28:06 [Parachain] PoV size { header: 0.4189453125kb, extrinsics: 10.9970703125kb, storage_proof: 29.11328125kb }
2025-01-14 10:28:06 [Parachain] Compressed PoV size: 36.8837890625kb
2025-01-14 10:28:06 [Relaychain] Trying to remove unknown reserved node 12D3KooWSChtDDwFkfBEwo7vQuyX4gX5ZheesUDsVCEotH5Mcj3K from SetId(3).
2025-01-14 10:28:06 [Relaychain] Trying to remove unknown reserved node 12D3KooWKPHKs8ZhDvK7NBYwgZ1JLyBaYUxkpvnA9zGgF2Pnfzzi from SetId(3).
2025-01-14 10:28:06 [Relaychain] Trying to remove unknown reserved node 12D3KooWHkp51QVBwvdoSKt2VZwApGymVZe4NsfZ8yd6PVDHnfos from SetId(3).
2025-01-14 10:28:06 [Relaychain] Trying to remove unknown reserved node 12D3KooWCc2tTgN1RutiN4uXHdS6qjzMQAn7ntNXAHzZoBSGAHmX from SetId(3).
2025-01-14 10:28:06 [Relaychain] Trying to remove unknown reserved node 12D3KooWSjWiGFr5ere6LDCyy1hYZoWNTkV8JCeF1bPWWJ4ySmQi from SetId(3).
2025-01-14 10:28:06 [Relaychain] ♻️  Reorg on #24282580,0xeada…2afe to #24282580,0x6cc5…6538, common ancestor #24282579,0x91ee…4d6c
2025-01-14 10:28:06 [Relaychain] 🏆 Imported #24282580 (0x91ee…4d6c → 0x6cc5…6538)
2025-01-14 10:28:06 [Parachain] 🙌 Starting consensus session on top of parent 0x07cf9cc6a987f9854823ce2bcc7dfc0435614f158f78aca00a3e5724295331e9
2025-01-14 10:28:06 [Parachain] 🎁 Prepared block for proposing at 7514442 (15 ms) [hash: 0x210b2a72092f42c37a5f42f350d9580fab245cdfd278b48163c8cfc73399dc7a; parent_hash: 0x07cf…31e9; extrinsics (3): [0xffdf…bf5e, 0x7317…2651, 0x791a…f793]
2025-01-14 10:28:06 [Parachain] 🔖 Pre-sealed block for proposal at 7514442. Hash now 0xb8d6cb5f77f105a5bbbdbfb8f37c08bc3f61a7faec1c0e57464f81ebb963c8aa, previously 0x210b2a72092f42c37a5f42f350d9580fab245cdfd278b48163c8cfc73399dc7a.
2025-01-14 10:28:06 [Parachain] 🆕 Imported #7514442 (0x07cf…31e9 → 0xb8d6…c8aa)
2025-01-14 10:28:06 [Parachain] PoV size { header: 0.2939453125kb, extrinsics: 11.4306640625kb, storage_proof: 144.255859375kb }
2025-01-14 10:28:06 [Parachain] Compressed PoV size: 121.6455078125kb
2025-01-14 10:28:06 [Relaychain] 💤 Idle (8 peers), best: #24282580 (0x6cc5…6538), finalized #24282577 (0x4c26…f6d6), ⬇ 202.4kiB/s ⬆ 500.5kiB/s
2025-01-14 10:28:06 [Parachain] 💤 Idle (15 peers), best: #7514442 (0xe038…e0d7), finalized #7514436 (0x449b…d2fa), ⬇ 5.6kiB/s ⬆ 103.2kiB/s
2025-01-14 10:28:11 [Relaychain] 💤 Idle (8 peers), best: #24282580 (0x6cc5…6538), finalized #24282577 (0x4c26…f6d6), ⬇ 41.2kiB/s ⬆ 394.4kiB/s
2025-01-14 10:28:11 [Parachain] 💤 Idle (15 peers), best: #7514442 (0xe038…e0d7), finalized #7514436 (0x449b…d2fa), ⬇ 28.5kiB/s ⬆ 29.8kiB/s
2025-01-14 10:28:12 [Relaychain] 🏆 Imported #24282581 (0x6cc5…6538 → 0xa304…0320)
2025-01-14 10:28:12 [Relaychain] ♻️  Reorg on #24282581,0xa304…0320 to #24282581,0x7480…2753, common ancestor #24282580,0x6cc5…6538
2025-01-14 10:28:12 [Relaychain] 🏆 Imported #24282581 (0x6cc5…6538 → 0x7480…2753)
2025-01-14 10:28:12 [Parachain] ♻️  Reorg on #7514442,0xe038…e0d7 to #7514443,0xe3d3…16e8, common ancestor #7514441,0x07cf…31e9
2025-01-14 10:28:12 [Parachain] 🏆 Imported #7514443 (0xb8d6…c8aa → 0xe3d3…16e8)
2025-01-14 10:28:12 [Parachain] 🆕 Imported #7514443 (0xb8d6…c8aa → 0x2934…f14a)
2025-01-14 10:28:16 [Parachain] 💤 Idle (15 peers), best: #7514443 (0xe3d3…16e8), finalized #7514436 (0x449b…d2fa), ⬇ 155.8kiB/s ⬆ 23.8kiB/s
2025-01-14 10:28:16 [Relaychain] 💤 Idle (8 peers), best: #24282581 (0x7480…2753), finalized #24282577 (0x4c26…f6d6), ⬇ 126.8kiB/s ⬆ 468.8kiB/s

Steps to reproduce

No response

[bkchr]

Collators produce a block, right after that, they drop it and collate a new one on its spot.
It seems reorgs coincide with forks on the relay chain.

Which collator are you using? The lookahead collator?

Generally forks are following the relay chain forks, as you already have assessed correctly. So, for each relay chain fork, there exists one parachain fork.

Did you generally run into any problems or just observed forks?

[Mihajlo-Pavlovic]

Yes, we are using lookahead collator.

I’ve noticed that reorgs occur approximately every 7 blocks, which is problematic for our use case. Since our application relies on minting NFTs, a reorg invalidates previously returned IDs, leading to significant issues. While waiting for finalization could mitigate this, the delay is often too long for our app's requirements.

I also observed that when a block is discarded during a reorg, a new block is created and finalized afterward. Is it possible to configure the collator to minimize reorgs? For example, could we delay block collation to allow more time for relay chain consensus, given that consensus seems to be reached relatively quickly?

[bkchr]

Are you using the omni-node? If yes, you could try to run with the --experimental-use-slot-based cli flag. It should lead to less forks, but doesn't prevent them totally.

@skunert we should probably expose the option to build on older relay chain parents, which would make it possible to almost prevent forks to interfere with the block production.