query commands optimization (Performance high memory usage)

[CarlosLopezDeLara]

What

Code review (joint effort ledger and cli) aiming to reduce memory usage of these commands

We have found that the original implementation was downloading the entire ledger state.
This is being fixed.

	Estimation
[ ] query stake-snapshot --> #4179	2
[ ] query leadership-schedule --> Merge IntersectMBO/ouroboros-network#3932 before --> #4250	-
[x] query pool-params --> #4170	done
[x] query ledger-state --> #4205	done

Why

The above are useful commands for stake pools but running them requires from 16 to 60GB of RAM. Crash the node and OOM any machine with less than that.

Benefits by persona

Stake pools: Can query their nodes reliably and get the information they need to make informed maintenance and business decisions.

Dependencies

None remaining.

Risks

None.

[gufmar]

This is definitely an increasingly needed improvement.
May also look at https://github.com/cardano-foundation/CIPs/blob/c39e55caff909e71efcb5911d62690e2e0c18118/cip-ledger-state.md

[AndrewWestberg]

I would also say that individually querying for each of the MARK, SET, and GO snapshots should be added.

[newhoggy]

I would also say that individually querying for each of the MARK, SET, and GO snapshots should be added.

Can you elaborate what being able to query for these separately gives us? Is this for performance reasons?

[AndrewWestberg]

@newhoggy Each of these snapshots is HUGE on its own. Being able to query just for the one you need greatly improves performance of utilizing them and parsing the cbor. It reduces memory and cpu usage.

[newhoggy]

This shouldn't be necessary. There will be a new query GetPoolDistr which will calculate the pool distributions server-side: https://github.com/input-output-hk/ouroboros-network/blob/master/ouroboros-consensus-shelley/src/Ouroboros/Consensus/Shelley/Ledger/Query.hs#L339-L341

[AndrewWestberg]

@newhoggy does that return the pool's sigma value for all 3 snapshots?

[newhoggy]

This is what you get:

$ cardano-cli query stake-snapshot --cardano-mode --mainnet --stake-pool-id fc79a939d5986c6565fc5e21bb5dd292261ac5da564c30385b3bb8d4

{
    "activeStakeGo": 24456415918186222,
    "activeStakeMark": 24453012636354912,
    "activeStakeSet": 24441961174746539,
    "poolStakeGo": 8595693051,
    "poolStakeMark": 8595693051,
    "poolStakeSet": 8595693051
}

I believe you can compute your sigma values from those.

There is also a PR to allows multiple stake pools to be queried at the same time which will change it to this:

#4279

[AndrewWestberg]

@newhoggy This command internally pulls down the entire ledger state from cardano-node to cardano-cli to calculate these values. It does not solve the high memory usage issue.