Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use 64-bit wide prefix rather than 10 decimal digits #104

Closed
silasdavis opened this issue Aug 20, 2018 · 7 comments
Closed

Use 64-bit wide prefix rather than 10 decimal digits #104

silasdavis opened this issue Aug 20, 2018 · 7 comments
Labels
T:enhancement Type: Enhancement

Comments

@silasdavis
Copy link
Contributor

It would take a few hundred years to cause my use case (one version per Tendermint block) any problems, but it currently we overflow various of our nodedb prefixes well before we overflow the underyling int64 type used for versions:

https://github.com/tendermint/iavl/blob/master/nodedb.go#L24-L30

IAVL used to load every node from previous versions:

v0.9.2...v0.10.0#diff-6d87913bee304887177dad726b816faeR256

Once we get past version 9999999999 then we will not be able to load newer versions because of the behaviour of Sscanf: https://github.com/tendermint/iavl/blob/master/nodedb.go#L344.

We'll actually still save them because the corresponding Sprintf will include digits outside the padding. But we'll hit this error when loading a large version: https://github.com/tendermint/iavl/blob/master/mutable_tree.go#L248

It would make sense to use a prefix that is 8 bytes - we could use raw big-endian padded bytes or we could use 16 chars of hex so that we maintain the lexicographical sorting we need for iteration. At least things blow up at the same time (and a significantly larger number) as well as it being neater to concede the same version bound.

Note the lazy-loading added by 0.10.0 makes it cheap to store a large number of versions, which is nice.

@ValarDragon
Copy link
Contributor

I totally agree, I find enforcing a specific number of digits quite strange, I definitely prefer this happening based on number of bits

@silasdavis
Copy link
Contributor Author

Would be nice to get this in sooner rather than later too since it is a breaking change.

I am happy to pick this up.

@ValarDragon
Copy link
Contributor

Please do! PR's are always appreciated

@liamsi
Copy link
Contributor

liamsi commented Aug 28, 2018

I also agree. Thanks @silasdavis!

@silasdavis
Copy link
Contributor Author

silasdavis commented Aug 30, 2018

@liamsi @ValarDragon cool please see my PR: #107

Do you think there is any chance getting this in a release next week? If there's a chance I might hold out releasing on my end.

@ValarDragon
Copy link
Contributor

ValarDragon commented Aug 30, 2018

I think we should make a minor release and include this in the next Tendermint. (This would be a breaking change) What are your thoughts @liamsi? I think the changes here can affect system performance on the whole, hence worth the inclusion in the next long running testnet.

@ValarDragon
Copy link
Contributor

This in the latest release! Thanks so much for doing this @silasdavis :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T:enhancement Type: Enhancement
Projects
None yet
Development

No branches or pull requests

3 participants