You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While testing Horizon 2.23.0 I decided to run verify-range tests because of the changes introduced in #4690. It seems that at the current ledger state size and full ledger this is no longer viable. Even after reverting on-disk mode (#4697) which makes CC much slower the m5.4xlarge box runs a realively small range of 10k ledgers from range [42667380, 43667380] in 5 hours!
Here are the ideas to make things faster:
Allow passing VERIFY_HISTORY variable to horizon-auto-verify-range pipeline. In some cases (like services/horizon: Improve performance of claimable balances queries #4690) we can skip history checks and it will make the tests overall time considerably faster. A PR for this has been opened in an internal repo.
Add a special flag to horizon ingest verify-range command that will skip history table updates and ingest only state changes (kind of opposite of horizon ingest reingest range command). Then we can move postgres data directory to a in-memory linux partition (currently the state tables and indexes occupy around 26 GB) that will make a) ingesting state, b) updating state, c) verifying state much faster. This solution may need an instance type with more ram than m5.4xlarge but it will probably consume less money because it will run much faster.
The text was updated successfully, but these errors were encountered:
While testing Horizon 2.23.0 I decided to run
verify-range
tests because of the changes introduced in #4690. It seems that at the current ledger state size and full ledger this is no longer viable. Even after reverting on-disk mode (#4697) which makes CC much slower them5.4xlarge
box runs a realively small range of 10k ledgers from range[42667380, 43667380]
in 5 hours!Here are the ideas to make things faster:
VERIFY_HISTORY
variable tohorizon-auto-verify-range
pipeline. In some cases (like services/horizon: Improve performance of claimable balances queries #4690) we can skip history checks and it will make the tests overall time considerably faster. A PR for this has been opened in an internal repo.horizon ingest verify-range
command that will skip history table updates and ingest only state changes (kind of opposite ofhorizon ingest reingest range
command). Then we can move postgres data directory to a in-memory linux partition (currently the state tables and indexes occupy around 26 GB) that will make a) ingesting state, b) updating state, c) verifying state much faster. This solution may need an instance type with more ram thanm5.4xlarge
but it will probably consume less money because it will run much faster.The text was updated successfully, but these errors were encountered: