Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: update maxHeaderSize #1800

Closed
wants to merge 1 commit into from

Conversation

microyahoo
Copy link

@microyahoo microyahoo commented Aug 14, 2022

Based on pkg-constants, the maxHeaderSize should be 22=5+5+10+1+1, which is discussed on Why is the maxHeaderSize 21 and not 22?

Signed-off-by: Liang Zheng [email protected]

@CLAassistant
Copy link

CLAassistant commented Aug 14, 2022

CLA assistant check
All committers have signed the CLA.

@microyahoo
Copy link
Author

microyahoo commented Aug 29, 2022

@jarifibrahim @manishrjain @MichelDiz PTAL, thanks.

Copy link
Contributor

@jarifibrahim jarifibrahim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good but I think this might break existing badger DBs because we use this value to determine where to write the data.

We'll need to bump the magicVersion over here to prevent existing databases from breaking https://github.com/dgraph-io/badger/blob/2de177900e3ced14fdab91aa7c6a7e3e441f0192/manifest.go#L248

Also, once this PR is merged, we'll need a major version update.

@joshua-goldstein joshua-goldstein added the status/accepted We accept to investigate or work on it. label Nov 4, 2022
@joshua-goldstein joshua-goldstein self-requested a review February 6, 2023 21:49
@joshua-goldstein joshua-goldstein changed the base branch from master to main February 6, 2023 21:53
Copy link
Contributor

@joshua-goldstein joshua-goldstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. This does indeed look like a bug which could cause a panic in

func (h header) Encode(out []byte) int {
	out[0], out[1] = h.meta, h.userMeta
	index := 2
	index += binary.PutUvarint(out[index:], uint64(h.klen))
	index += binary.PutUvarint(out[index:], uint64(h.vlen))
	index += binary.PutUvarint(out[index:], h.expiresAt)
	return index
}

since the size of the out buffer is allocated based on this parameter. We haven't seen this issue before since it would require each varInt encoded field to be its maximum size, but it certainly could happen. This doesn't require a magic number upgrade since this is fixing an edge case.

@mangalaman93
Copy link
Member

@joshua-goldstein could you write a test for it so that in case something changes, the test fails. We also should figure out what are the cases when the header size could become 22.

Based on https://pkg.go.dev/encoding/binary#pkg-constants, the maxHeaderSize should be 22=5+5+10+1+1.

Signed-off-by: Liang Zheng <[email protected]>
@mangalaman93
Copy link
Member

I am closing this PR in lieu of #1877. Once again, thank you for your contribution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status/accepted We accept to investigate or work on it.
Development

Successfully merging this pull request may close these issues.

5 participants