_distributeExcessIdle overhaul

[jalextowle]

_distributeExcessIdle Algorithm

The _distributeExcessIdle function is responsible for paying out as many withdrawal shares as possible with the pool's idle liquidity. This boils down to finding the share proceeds $\Delta z$ and withdrawal shares redeemed $\Delta w$ such that the LP share price is held constant.

Notation

Assume that we're using the usual math notation unless specified otherwise. Functions of the form $f(x)$ are assumed to represent a quantity parameterized by the amount of shares $x$ that we are removing from the share reserves. We define some additional values to make our lives easier when we run through the calculations:

$I$: The pool's idle liquidity.

$l$: The total supply of LP shares (including active and outstanding withdrawal shares).

$w$: The supply of withdrawal shares that are outstanding (i.e. they haven't been marked as ready for withdrawal yet).

$y_l$: The amount of longs outstanding.

$y_s$: The amount of shorts outstanding.

$t_l$: The average normalized time remaining of longs.

$t_s$: The average normalized time remaining of shorts.

Problem Statement

We need to pay out as many withdrawal shares as possible while conserving the LP share price. This can be stated formally as:

$$ \begin{align} \max \quad & \Delta w \\ \text{s.t.} \quad & \frac{PV(0)}{l} = \frac{PV(\Delta z)}{l - \Delta w} \\ & \Delta z \leq I \\ & \Delta w \leq w \\ & y_s \cdot t_s - y_l \cdot t_l \leq y_{out}^{max}(\Delta z) \end{align} $$

We can solve this problem by breaking it down into two cases:

Case 1: $y_s \cdot t_s \leq y_l \cdot t_l$

In this case, the pool either completely nets out or it is net long. In this situation, we can drop the last constraint since for all $\Delta z \geq 0$ we have $y_{out}^{max}(\Delta z) \geq 0$. We can break this down into two sub-cases:

Case 1.1: $\Delta z = I$

All of the idle can be paid out. We can solve $\tfrac{PV(0)}{l} = \tfrac{PV(I)}{l - \Delta w}$ for $\Delta w$ to get: $\Delta w = \left(1 - \tfrac{PV(I)}{PV(0)} \right) \cdot l$.

Case 1.2: $\Delta w = w$

All of the withdrawal shares can be paid out. We solve $\tfrac{PV(0)}{l} = \tfrac{PV(\Delta z)}{l - w}$ for $\Delta z$. Unfortunately, YieldSpace makes it hard or impossible to solve directly in this situation. Fortunately, the present value is differentiable, so we can use an iterative method like Newton's method to find a suitably close approximation to $\Delta z$.

Our objective function $F(\Delta z)$ is given by:

$$ F(\Delta z) = PV(\Delta z) \cdot l - PV(0) \cdot (l - w). $$

The derivative $F'(\Delta z)$ is given by:

$$ F'(\Delta z) = PV'(\Delta z) \cdot l $$

Assuming we have an initial guess $\Delta z_0$ of $\Delta z$, we can calculate the $n$th, $n \geq 1$, step $\Delta z$ of Newton's Method as:

$$ \Delta z_n = \Delta z_{n - 1} - \frac{F(\Delta z_{n - 1})}{F'(\Delta z_{n - 1})} $$

Case 2: $y_s \cdot t_s > y_l \cdot t_l$

Since the pool is net short, the last constraint is actually relevant. We can start by checking if $y_{out}^{max}(I) \geq y_s \cdot t_s - y_l \cdot t_l$. If this condition holds, then we aren't constrained by the maximum amount of bonds out, and we can proceed by working through cases 1.1 and 1.2. Otherwise, we need to find the upper bound $\Delta z_{max}$ such that $y_{out}^{max}(\Delta z_{max}) = y_s \cdot t_s - y_l \cdot t_l$. Even though we can't solve for this analytically, we can solve this using Newton's method. Once we have the value $\Delta z_{max}$, we can substitute this value in for $I$ in Case 1.1 and 1.2 and proceed normally from there.

Algorithm

1. If $y_s \cdot t_s \leq y_l \cdot t_l$ or $y_{out}^{max}(I) \geq y_s \cdot t_s - y_l \cdot t_l$, set $\Delta z_{max} = I$ and proceed to step (3). Otherwise, proceed to step (2).
2. Solve $y_{out}^{max}(\Delta z_{max}) = y_s \cdot t_s - y_l \cdot t_l$ for $\Delta z_{max}$ using Newton's method.
3. Set $\Delta w = \left(1 - \tfrac{PV(\Delta z_{max})}{PV(0)} \right) \cdot l$. If $\Delta w \leq w$, then proceed to step (5). Otherwise, continue to step (4).
4. Set $\Delta w = w$ and solve $\tfrac{PV(0)}{l} = \tfrac{PV(\Delta z)}{l - \Delta w}$ for $\Delta z$ using Newton's method if $y_l \cdot t_l \neq y_s \cdot t_s$ or directly otherwise.
5. Mark $\Delta w$ withdrawal shares as ready for withdrawal. Pay out $\Delta z$ shares to the withdrawal pool and reduce the share reserves by $\Delta z$.

Appendix

Identities

We define the follow identities for convenience:

1. $z_e = z - \zeta$
2. $z(x) = z - x$
   • $z'(x) = -1$.
3. $\zeta(x) = \zeta \cdot \tfrac{z(x)}{z} = \tfrac{\zeta}{z} \cdot \left( z - x \right)$
   • $\zeta'(x) = - \tfrac{\zeta}{z}$
4. $z_e(x) = z(x) - \zeta(x) = \left( 1 - \tfrac{\zeta}{z} \right) \cdot \left( z - x \right)$
   • $z_e'(x) = - \left(1 - \tfrac{\zeta}{z} \right)$
5. $y(x) = y \cdot \tfrac{z_e(x)}{z_e} = \tfrac{y}{z_e} \cdot \left( 1 - \tfrac{\zeta}{z} \right) \cdot \left( z - x \right)$
   • $y'(x) = - \tfrac{y}{z_e} \cdot \left( 1 - \tfrac{\zeta}{z} \right)$
6. $k(x) = \tfrac{c}{\mu} \cdot \left( \mu \cdot z_e(x) \right)^{1 - t_s} + y(x)^{1 - t_s}$
   • $k'(x) = - \left( 1 - t_s \right) \cdot \left( 1 - \tfrac{\zeta}{z} \right) \cdot \left( c \cdot (\mu \cdot z_e(x))^{-t_s} + \tfrac{y}{z_e} \cdot (y(x))^{-t_s} \right)$

Yield Space Math

Buying Bonds

We can calculate the amount of shares in $z_{in}(x, \Delta y)$ given the amount of bonds out $\Delta y$ as:

$$ z_{in}(x, \Delta y) = \tfrac{1}{\mu} \cdot \left( \tfrac{\mu}{c} \cdot \left( k(x) - \left( y(x) - \Delta y \right)^{1 - t_s} \right) \right)^{\tfrac{1}{1 - t_s}} - z_e(x). $$

We can calculate the derivative $z_{in}'(x, \Delta y)$ as:

$$ \begin{align} z_{in}'(x, \Delta y) & = \tfrac{1}{\mu} \cdot \tfrac{\mu}{c} \cdot \tfrac{1}{1 - t_s} \cdot \left( k'(x) - (1 - t_s) \cdot y'(x) \cdot \left( y(x) - \Delta y \right)^{-t_s} \right) \cdot \left( \tfrac{\mu}{c} \cdot \left( k(x) - \left( y(x) - \Delta y \right)^{1 - t_s} \right) \right)^{\tfrac{t_s}{1 - t_s}} - z_e'(x) \\ & = - \left( 1 - \tfrac{\zeta}{z} \right) \cdot \left( \tfrac{1}{c} \cdot \left( c \cdot \left( \mu \cdot z_e(x) \right)^{-t_s} + \tfrac{y}{z_e} \cdot \left( y(x) \right)^{-t_s} - \tfrac{y}{z_e} \cdot \left( y(x) - \Delta y \right)^{-t_s} \right) \cdot \left( \tfrac{\mu}{c} \cdot \left( k(x) - \left( y(x) - \Delta y \right)^{1 - t_s} \right) \right)^{\tfrac{t_s}{1 - t_s}} - 1 \right). \end{align} $$

We can calculate the max buy $y_{out}^{max}(x)$ on Yield Space as:

$$ y_{out}^{max}(x) = y(x) - \left( \frac{k(x)}{\tfrac{c}{\mu} + 1} \right)^{\tfrac{1}{1 - t_s}}. $$

We can calculate the derivative $y_{out}^{max'}(x)$ as:

$$ \begin{align} y_{out}^{max'}(x) & = y'(x) - \frac{1}{1 - t_s} \cdot \frac{1}{\tfrac{c}{\mu} + 1} \cdot k'(x) \cdot \left( \frac{k(x)}{\tfrac{c}{\mu} + 1} \right)^{\tfrac{t_s}{1 - t_s}} \\ & = - \frac{y}{z_e} \cdot \left( 1 - \frac{\zeta}{z} \right) - \frac{1}{1 - t_s} \cdot \frac{1}{ \tfrac{c}{\mu} + 1 } \cdot \left( - \left( 1 - t_s \right) \right) \cdot \left( 1 - \tfrac{\zeta}{z} \right) \cdot \left( c \cdot \left( \mu \cdot z_e(x) \right)^{-t_s} + \tfrac{y}{z_e} \cdot \left( y(x) \right)^{-t_s} \right) \cdot \left( \frac{k(x)}{\tfrac{c}{\mu} + 1} \right)^{\tfrac{t_s}{1 - t_s}} \\ & = - \left( 1 - \frac{\zeta}{z} \right) \cdot \left( \frac{y}{z_e} - \frac{1}{\tfrac{c}{\mu} + 1} \cdot \left( c \cdot \left( \mu \cdot z_e(x) \right)^{-t_s} + \frac{y}{z_e} \cdot \left( y(x) \right)^{-t_s} \right) \cdot \left( \frac{k(x)}{\tfrac{c}{\mu} + 1} \right)^{\tfrac{t_s}{1 - t_s}} \right) \end{align} $$

We can calculate the share payment required to buy the maximum amount of bonds $z_{in}^{max}(x)$ as $z_{in}^{max}(x) = z_{in}(x, y_{out}^{max}(x))$; however, we can calculate this more directly since we know that $p = \left(\tfrac{\mu \cdot z_{in}^{max}(x)}{y_{out}^{max}(x)}\right)^{t_s} = 1 \implies \mu \cdot z_{in}^{max}(x) = y_{out}^{max}(x)$:

$$ z_{in}^{max}(x) = \tfrac{1}{\mu} \cdot y_{out}^{max}(x). $$

We can calculate the derivative $z_{in}^{max'}(x)$ as:

$$ z_{in}^{max'}(x) = \tfrac{1}{\mu} \cdot y_{out}^{max'}(x). $$

Selling Bonds

We can calculate the amount of shares out $z_{out}(x, \Delta y)$ given the amount of bonds in $\Delta y$ as:

$$ z_{out}(x, \Delta y) = z_e(x) - \tfrac{1}{\mu} \cdot \left( \tfrac{\mu}{c} \cdot \left( k(x) - (y(x) + \Delta y)^{1 - t_s} \right) \right)^{\tfrac{1}{1 - t_s}}. $$

We can calculate the derivative $z_{out}'(x)$ as:

$$ \begin{align} z'_{out}(x, \Delta y) & = z_e'(x) - \tfrac{1}{\mu} \cdot \left[ \tfrac{1}{1 - t_s} \cdot \tfrac{\mu}{c} \cdot \left( k'(x) - (1 - t_s) \cdot y'(x) \cdot \left( y(x) + \Delta y \right)^{-t_s} \right) \cdot \left( \tfrac{\mu}{c} \cdot \left( k(x) - \left( y(x) + \Delta y \right)^{1 - t_s} \right) \right)^{\tfrac{t_s}{1 - t_s}} \right] \\ & = \left( \tfrac{\zeta}{z} - 1 \right) - \tfrac{1}{c} \cdot \tfrac{1}{1 - t_s} \cdot (1 - t_s) \cdot \left( \tfrac{\zeta}{z} - 1 \right) \cdot \left( c \cdot (\mu \cdot z_e(x))^{-t_s} + \tfrac{y}{z_e} \cdot \left( y(x) \right)^{-t_s} - \tfrac{y}{z_e} \cdot \left( y(x) + \Delta y \right)^{-t_s} \right) \cdot \left( \tfrac{\mu}{c} \cdot \left( k(x) - \left( y(x) + \Delta y \right)^{1 - t_s} \right) \right)^{\tfrac{t_s}{1 - t_s}} \\ & = - \left( 1 - \tfrac{\zeta}{z} \right) \cdot \left( 1 - \tfrac{1}{c} \cdot \left( c \cdot (\mu \cdot z_e(x))^{-t_s} + \tfrac{y}{z_e} \cdot \left( y(x) \right)^{-t_s} - \tfrac{y}{z_e} \cdot \left( y(x) + \Delta y \right)^{-t_s} \right) \cdot \left( \tfrac{\mu}{c} \cdot \left( k(x) - \left( y(x) + \Delta y \right)^{1 - t_s} \right) \right)^{\tfrac{t_s}{1 - t_s}} \right). \end{align} $$

FIXME: This is wrong when $\zeta < 0$. We've updated this in the code, but it would be good to call this out here. We also don't need the derivative, and we should mention that we don't need it when we forego adding it here.

We can calculate the maximum amount of bonds that can be sold as $y_{in}^{max}(x)$ on Yield Space as:

$$ y_{in}^{max}(x) = \left( k(x) - \tfrac{c}{\mu} \cdot \left( \mu \cdot z_{min} \right)^{1 - t_s} \right)^{\tfrac{1}{1 - t_s}} - y(x). $$

NOTE: This is how $y_{in}^{max}(x)$ is calculated in YieldSpaceMath.sol; however, we don't enforce that $z - \zeta \geq z_{min}$ as of right now. We're going to make this change, so we can proceed assuming that was already fixed.

We can calculate the derivative $y_{in}^{max'}(x)$ as:

$$ \begin{align} y_{in}^{max'}(x) & = \tfrac{1}{1 - t_s} \cdot k'(x) \cdot \left( k(x) - \tfrac{c}{\mu} \cdot ( \mu \cdot z_{min} )^{1 - t_s} \right)^{\tfrac{t_s}{1 - t_s}} - \tfrac{y}{z_e} \cdot \left( \tfrac{\zeta}{z} - 1 \right) \\ & = \left( \tfrac{\zeta}{z} - 1 \right) \cdot \left( \left( c \cdot ( \mu \cdot z_e(x) )^{-t_s} + \tfrac{y}{z_e} \cdot ( y(x) )^{-t_s} \right) \cdot \left( k(x) - \tfrac{c}{\mu} \cdot (\mu \cdot z_{min})^{1 - t_s} \right)^{\tfrac{t_s}{1 - t_s}} - \tfrac{y}{z_e} \right). \end{align} $$

We can calculate the share proceeds of selling the maximum amount of bonds $z_{out}^{max}(x)$ as $z_{out}^{max}(x) = z_{out}(x, y_{in}^{max}(x))$; however, we know that the minimum effective share reserves is $z_{min}$, so we can quickly calculate this as:

$$ z_{out}^{max}(x) = z_e(x) - z_{min}. $$

We can calculate the derivative $z_{out}^{max'}(x)$ as:

$$ z_{out}^{max'}(x) = z_e'(x) = - \left( 1 - \tfrac{\zeta}{z} \right). $$

*From this, the optimization problem is trivial when we get into the regime where the net curve position can't be closed on the curve. We should make sure to test this edge-case.

Present Value Math

Present Value Function

Using the notation we've defined, we can rigorously define the LP present value as a function of the amount of shares $x$ that we remove from the share reserves to pay out the withdrawal pool. At a high-level we can define $PV(x)$ as:

$$ PV(x) = z(x) + net_c(x) + net_f - z_{min}. $$

$net_c(x)$ is the impact to the share reserves of closing the pool's net curve position. Similarly, $net_f$ is the impact to the share reserves of closing the pool's net flat position. Since the payouts are fixed at maturity, $net_f$ is a constant with respect to $x$. We can define $net_f$ as:

$$ net_f = \frac{y_s \cdot (1 - t_s) - y_l \cdot (1 - t_l)}{c}. $$

We can define $net_c(x)$ as a piecewise function as follows:

$$ net_c(x) = \left{\begin{array}{lr} net_c^l(x), & \text{if } y_l \cdot t_l > y_s \cdot t_s\\ net_c^s(x), & \text{if } y_s \cdot t_s > y_l \cdot t_l \\ 0 & \text{if } y_l \cdot t_l = y_s \cdot t_s \end{array}\right. $$

$y_l \cdot t_l > y_s \cdot t_s$ means that the pool is net long, and $y_s \cdot t_s > y_l \cdot t_l$ means that the pool is net short. $net_c^l(x)$ and $net_c^s(x)$ are the variants of the net curve calculation that pertain to longs and shorts, respectively.

We calculate $net_c^{l}(x)$ by attempting to close as much of the net long position as possible. We mark any remaining longs in the net position to zero because the pool's spot price is 0 after making the max sell:

$$ net_c^{l}(x) = \left{\begin{array}{lr} -z_{out}^{max}(x), & \text{if } y_l \cdot t_l - y_s \cdot t_s > y_{in}^{max}(x) \\ -z_{out}(x, y_l \cdot t_l - y_s \cdot t_s) & \text{if } y_l \cdot t_l - y_s \cdot t_s \leq y_{in}^{max}(x) \end{array}\right. $$

We calculate $net_c^{s}(x)$ by attempting to close as much of the net short position as possible. We mark any remaining shorts in the net position to 1 because the pool's spot price is 1 after making the max buy. Let $y_{net} = y_s \cdot t_s - y_l \cdot t_l$:

$$ net_c^{s}(x) = \left{\begin{array}{lr} z_{in}^{max}(x) + \left( \left( y_s \cdot t_s - y_l \cdot t_l \right) - y_{out}^{max}(x) \right), & \text{if } y_s \cdot t_s - y_l \cdot t_l > y_{out}^{max}(x) \\ z_{in}(x, y_s \cdot t_s - y_l \cdot t_l) & \text{if } y_s \cdot t_s - y_l \cdot t_l \leq y_{out}^{max}(x) \end{array}\right. $$

Present Value Derivative

We can calculate the derivative of the present value function at a high-level as:

$$ PV'(x) = -1 + net_c'(x). $$

The derivative $net_c'(x)$ is calculated component-wise as:

$$ net_c'(x) = \left{\begin{array}{lr} net_c^{l'}(x), & \text{if } y_l \cdot t_s > y_s \cdot t_s \\ net_c^{s'}(x), & \text{if } y_s \cdot t_s < y_l \cdot t_l \\ 0 & \text{if } y_l \cdot t_l = y_s \cdot t_s \end{array}\right. $$

The derivative $net_c^{l'}(x)$ is calculated component-wise as:

$$ net_c^{l'}(x) = \left{\begin{array}{lr} -z_{out}^{max'}(x), & \text{if } y_l \cdot t_l - y_s \cdot t_s > y_{in}^{max}(x) \\ -z_{out}'(x, y_l \cdot t_l - y_s \cdot t_s), & \text{if } y_l \cdot t_l - y_s \cdot t_s \leq y_{in}^{max}(x) \end{array}\right. $$

The derivative $net_c^{s'}(x)$ is calculated component-wise as:

$$ net_c^{s'}(x) = \left{\begin{array}{lr} z_{in}^{max'}(x) - y_{out}^{max'}(x), & \text{if } y_s \cdot t_s - y_l \cdot t_l > y_{out}^{max}(x) \\ z_{in}'(x, y_s \cdot t_s - y_l \cdot t_l) & \text{if } y_s \cdot t_s - y_l \cdot t_l \leq y_{out}^{max}(x) \end{array}\right. $$

[github-actions]

Hyperdrive Gas Benchmark

Benchmark suite	Current: 0e7fbbe	Previous: 099d850	Deviation	Status
addLiquidity: min	1600 gas	1622 gas	-1.3564%	✅
addLiquidity: avg	52538 gas	54725 gas	-3.9963%	✅
addLiquidity: max	232092 gas	98648 gas	135.2729%	🚨
checkpoint: min	1216 gas	1216 gas	0%	🟰
checkpoint: avg	47997 gas	48259 gas	-0.5429%	✅
checkpoint: max	201955 gas	97592 gas	106.9381%	🚨
closeLong: min	1690 gas	1690 gas	0%	🟰
closeLong: avg	26524 gas	24890 gas	6.5649%	🚨
closeLong: max	139742 gas	114354 gas	22.2012%	🚨
closeShort: min	1693 gas	1693 gas	0%	🟰
closeShort: avg	28836 gas	27649 gas	4.2931%	🚨
closeShort: max	114723 gas	108976 gas	5.2736%	🚨
initialize: min	1605 gas	1605 gas	0%	🟰
initialize: avg	181825 gas	179780 gas	1.1375%	🚨
initialize: max	256942 gas	254322 gas	1.0302%	🚨
openLong: min	736 gas	736 gas	0%	🟰
openLong: avg	39667 gas	56291 gas	-29.5323%	✅
openLong: max	161281 gas	196013 gas	-17.7192%	✅
openShort: min	702 gas	702 gas	0%	🟰
openShort: avg	46283 gas	55560 gas	-16.6973%	✅
openShort: max	160559 gas	194932 gas	-17.6333%	✅
redeemWithdrawalShares: min	1598 gas	1598 gas	0%	🟰
redeemWithdrawalShares: avg	17371 gas	22367 gas	-22.3365%	✅
redeemWithdrawalShares: max	80714 gas	49853 gas	61.9040%	🚨
removeLiquidity: min	1661 gas	1661 gas	0%	🟰
removeLiquidity: avg	119560 gas	77183 gas	54.9046%	🚨
removeLiquidity: max	258350 gas	204563 gas	26.2936%	🚨

This comment was automatically generated by workflow using github-action-benchmark.

[coveralls]

coverage: 95.366% (-0.4%) from 95.788%
when pulling 8319231 on jalextowle/feature/distribute-excess-idle-overhaul-2
into 2a51316 on main.

[jrhea]

does this PR address #588 ?