Gas stability pool mechanism improvement

[lumtis]

Extension of #3221, this issue is for a bug while this one is for a general optimization of the stability pool mechanism (eventually both are completed in the same PR)

Current mechanism

• For each connected chains, a "gas stability pool" exists that is funded manually and with remaining fees from previous transactions
• We store in the blockchain "gas stability flags" that describes how gas price must be increased with gas stability pool
• At each epoch (configurable number of blocks), the pending cctxs are iterated and depending for how long it has been pending, the gas price is increased by a percentage from the flags, there is a percentage limit for max gas price. The difference of gas limit is funded from gas stability pool (then unused fees are resent unto the pool)

Some issues with the current mechanism

• The increase is a fix percentage in the flags, independent of the gas price hikes, for very big hike in the price, the flags might be ineffective, if the flags are set to high percentage then it is not optimized for smaller gas price hike
• Multiple outbounds are increased at the same time, outbound+100 might be increased while outbound+1 is blocked, all the pending outbound are increased, the gas price might again fall once the blocked outbound get signed, this might end up in a big waste of funds with unnecessary high gas price for most outbounds
• The current mechanism is not adapted for EIP1559 since the increase for gasFeeCap and priorityFee might not be the same

Proposed solution

• Update with more accurate values: remove the gas increase percent flag, instead when iterating pending outbound, the increase is determined by the current voted gas price. The difference in the fee limit is used for the gas stability pool. For EIP1559 both gasFeeCap and priorityFee are updated with the voted values
• Having less look ahead and bigger focus on currently blocked outbound. Instead of iterating n+x next outbound, only the n+1 pending outbound is parsed (we can maybe tolerate a few ore but idea is to have a very small look ahead), it is parsed more frequently (smaller epoch), the gas price is updated if any of two conditions are met (been pending for a x delay OR the current voted gas price is above a threshold of the gas price currently used). If the gas price is increased while the tx has been signed and is being confirmed, this is not an issue as the effective fee paid will be used to refund to the gas stability pool.

In short the gas stability mechanism is more concentrated to the currently blocked outbound. There will be slower time to update gas price of all outbound in gas of dramatic hike in the price and large number of outbound in a short time frame but use of the gas stability pool will be made more optimized.

[ws4charlie]

After researching Bitcoin tx replacement, I noticed below differences:

• EVM chain transactions can be replaced separately. For example, a sequential transactions: tx1, tx2, tx3, tx4 got stuck in mempool. The TSS signers can replace each one of them separately by increasing fees.

• Bitcoin transactions can not be replaced separately. For example, a sequential transactions: tx1, tx2, tx3, tx4 got stuck in mempool. If the TSS signers replace any of of the txs, for instance tx2, all subsequent txs (here is tx3, tx4) will become invalid and evicted from mempool. TSS signers has to re-construct and re-sign tx3, tx4.

After seeing the difference mechanism of tx replacements, I was asking myself: is there a universal way to achieve tx replacement? maybe not?

Here were more thoughts afterwards:

• zetacore bumps cctx gas price every 10 minutes and burns gas token, hoping that zetaclient uses the bumped fee to re-broadcasted the outbound.
• there is no guarantee that zetaclient will replace the tx with updated fee, because a pending tx can still finalize before zetaclient effectively take action, and the updated gas price is then just a suggestion.
• Bitcoin block time is 10 minute, so it usually takes >10 minutes (or >1 block) to be able to tell if a pending tx is stuck or not. The sensitivity that we response to pending tx leads to different tx replacement timing.
• zetacore has no idea of what's happening in external chains (RPC error, outage?) and just blindly burns fee (from stability pool) to funds the pending CCTXs (> 10 minutes), hoping zetaclient to push the tx through.

In the Bitcoin RBF transaction PR (in progress and tests), I created a mempool monitoring goroutine to watch and detect all pending transactions in mempool to determine stuck/non-stuck status and decide replacing or not.

[kingpinXD]

Based on the conversation we had and after some research .
I think some points from my end are

zetacore has no idea of what's happening in external chains (RPC error, outage?) and just blindly burns fee (from stability pool) to funds the pending CCTXs (> 10 minutes), hoping zetaclient to push the tx through.

This is true, but Zetacore expects the clients to provide this information through the outbound Vote, which can then be used to adjust balances.

• When the BTC tx is confirmed, we should decide how funds get added to the stability pool (Extra tokens if any)

Also, I'm curious , since we are bumping up the latest pending CCTX, would that mean the entire pending queue gets cleared in one block if the fee supplied appeals to the miner?