Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use segregated hashmap to boost the freelist allocate and release performance #141

Merged
merged 1 commit into from
Jan 25, 2019
Merged

use segregated hashmap to boost the freelist allocate and release performance #141

merged 1 commit into from
Jan 25, 2019

Conversation

WIZARD-CXY
Copy link
Contributor

@WIZARD-CXY WIZARD-CXY commented Jan 22, 2019 •
edited
Loading

In this pr, I use segregated hashmap to replace the original freeids allocating and releasing approach.
It is much faster than the original version, especially when the db size is large or the fragmentation in the db is large, we can gain 1000x faster performance.

@WIZARD-CXY WIZARD-CXY changed the title use segregated hashmap to boost the freelist allocate and release per… use segregated hashmap to boost the freelist allocate and release performance Jan 22, 2019
xiang90
xiang90 reviewed Jan 22, 2019
View reviewed changes
freelist.go Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 22, 2019
View reviewed changes
freelist.go Outdated Show resolved Hide resolved
@WIZARD-CXY WIZARD-CXY force-pushed the new branch 3 times, most recently from ed79fac to f6fcd69 Compare January 22, 2019 08:10
xiang90
xiang90 reviewed Jan 22, 2019
View reviewed changes
freelist_hmap.go Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 22, 2019
View reviewed changes
freelist_test.go Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 22, 2019
View reviewed changes
freelist.go Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 22, 2019
View reviewed changes
freelist.go Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 22, 2019
View reviewed changes
freelist.go Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 22, 2019
View reviewed changes
freelist.go Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 22, 2019
View reviewed changes
freelist_test.go Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 22, 2019
View reviewed changes
freelist_test.go Outdated Show resolved Hide resolved
@WIZARD-CXY WIZARD-CXY force-pushed the new branch 5 times, most recently from 902e89d to d73f0dc Compare January 23, 2019 02:59
xiang90
xiang90 reviewed Jan 23, 2019
View reviewed changes
Makefile Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 23, 2019
View reviewed changes
Makefile Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 24, 2019
View reviewed changes
freelist_test.go Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 24, 2019
View reviewed changes
db.go Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 24, 2019
View reviewed changes
db.go Outdated Show resolved Hide resolved
@WIZARD-CXY WIZARD-CXY force-pushed the new branch 2 times, most recently from 376ff9b to cec1f50 Compare January 24, 2019 04:18
@WIZARD-CXY
Copy link
Contributor Author

@xiang90 ptal

xiang90
xiang90 reviewed Jan 24, 2019
View reviewed changes
db_test.go Outdated Show resolved Hide resolved
@xiang90
Copy link
Contributor

xiang90 commented Jan 24, 2019

@WIZARD-CXY Awesome work!

can you ask @hormes to give this a final look (especially on the tests)?

xiang90
xiang90 reviewed Jan 24, 2019
View reviewed changes
db.go Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 24, 2019
View reviewed changes
db.go Show resolved Hide resolved
@xiang90
Copy link
Contributor

xiang90 commented Jan 24, 2019

/cc @jpbetz

xiang90
xiang90 reviewed Jan 24, 2019
View reviewed changes
freelist_test.go Outdated Show resolved Hide resolved
@WIZARD-CXY
Copy link
Contributor Author

For the compatibility reason, I keep all the old test which is used to test the old array freelist type and make them pass. The new approach is using the same test and make it fully passed.

@hormes
Copy link

hormes commented Jan 25, 2019

Everything looks good to me. @xiang90

xiang90
xiang90 reviewed Jan 25, 2019
View reviewed changes
db.go Outdated Show resolved Hide resolved
xiang90
xiang90 reviewed Jan 25, 2019
View reviewed changes
db.go Outdated Show resolved Hide resolved
@xiang90
Copy link
Contributor

xiang90 commented Jan 25, 2019

lgtm

@xiang90 xiang90 merged commit 8693da9 into etcd-io:master Jan 25, 2019
thaJeztah added a commit to thaJeztah/containerd that referenced this pull request Feb 21, 2019
@thaJeztah
Update go.etcd.io/bbolt to v1.3.2
cbc032e 
Relevant changes:

- ectd-io/bbolt#139 update the freelist readIDs
- etcd-io/bbolt#140 add getFreePageIDs
- etcd-io/bbolt#141 use segregated hashmap to boost the freelist allocate and release performance

Signed-off-by: Sebastiaan van Stijn <[email protected]>
@thaJeztah thaJeztah mentioned this pull request Feb 21, 2019
Merged
@Crypt-iQ Crypt-iQ mentioned this pull request Jun 24, 2019
Closed
@bbotte bbotte mentioned this pull request Aug 20, 2019
Closed
@ChinX ChinX mentioned this pull request Dec 9, 2019
Merged
6 tasks
@Doyle-net Doyle-net mentioned this pull request Jul 25, 2020
Closed
@Doyle-net
Copy link

In this pr, I use segregated hashmap to replace the original freeids allocating and releasing approach.
It is much faster than the original version, especially when the db size is large or the fragmentation in the db is large, we can gain 1000x faster performance.

Hello, may I ask
Does the new version of ETCD already have this feature?

@WIZARD-CXY
Copy link
Contributor Author

In this pr, I use segregated hashmap to replace the original freeids allocating and releasing approach.
It is much faster than the original version, especially when the db size is large or the fragmentation in the db is large, we can gain 1000x faster performance.

Hello, may I ask
Does the new version of ETCD already have this feature?

yeah

@yydzhou
Copy link

yydzhou commented Apr 21, 2022

Hi @WIZARD-CXY Thanks so much for the etcd performance optimization contribution.
Since you mentioned the new algorithm is in the new version of etcd, does that mean the description of "recommand 8GB" storage in this link is not truth anymore? https://etcd.io/docs/v3.5/dev-guide/limit/

@NsrWissam NsrWissam mentioned this pull request Jun 3, 2022
Open
@wanghenshui wanghenshui mentioned this pull request Feb 1, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiang90 xiang90 xiang90 left review comments

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@WIZARD-CXY @xiang90 @hormes @Doyle-net @yydzhou