fix(Buffer): uint32 -> uint64 in slice methods #323
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
I'm currently on Dgraph 21.12 unable to export my data.
Export fails on just two nodes out of 7 with the "Unexpected EOF" error:
Skipping rather long investigation of this issue I came to find length of slice, written to the Buffer during export exceed the size of uint32 (i've decoded varint before Value field in Badger KV struct with RDF's to get something around 4.5Gb, which is expected for a rather bloated reverse edge to the one of the most common nodes in my DB. Also count query returns 72 105 794 connected nodes which is, welp, quite a lot).
Not to mention that working with
int
which is almost always isint64
and then casually casting it touint32
w/o any checks or warnings is as bad as it gets.Solution
Find any
4
andUint32
and carefully replace them with8
andUint64
. As this happens only in slice-related methods the fix is quite easy. Locally tests run just fine, but i had to patch the sort one to accommodate for size changes. Also i did test 21.12-related badger version and tests run fine too.Afterword
I'm somewhat in a hurry, so sorry for not being able to provide more info. I'll write something here some time later.
Also, as far as a can tell - any version of dgraph should be affected by this. Not sure how I turned out to be the first one.