Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

internal/lz4block: Faster table reset #169

Merged
merged 2 commits into from
Feb 12, 2022
Merged

Conversation

greatroar
Copy link
Contributor

Zeroing out the 128kiB hash table takes a considerable amount of time in repeated calls to CompressBlock on small, poorly compressible blocks. Give the hashtable an 8kiB bitmap indicating which entries are in use and zero that out instead.

Linux/amd64, Go 1.17:

name              old speed      new speed      delta
CompressRandom-8  1.58GB/s ± 2%  1.94GB/s ± 2%  +23.44%  (p=0.000 n=20+19)
CompressPg1661-8   826MB/s ± 5%   837MB/s ± 2%     ~     (p=0.101 n=19+20)
CompressDigits-8   151MB/s ± 1%   150MB/s ± 2%   -0.73%  (p=0.012 n=20+20)
CompressTwain-8    558MB/s ± 2%   559MB/s ± 2%     ~     (p=1.000 n=20+20)
CompressRand-8    25.1MB/s ± 2%  25.1MB/s ± 3%     ~     (p=1.000 n=20+20)

Go 1.18beta:

name              old speed      new speed      delta
CompressRandom-8  1.58GB/s ± 1%  1.95GB/s ± 1%  +23.66%  (p=0.000 n=19+19)
CompressPg1661-8   765MB/s ± 4%   764MB/s ± 6%     ~     (p=0.678 n=20+20)
CompressDigits-8   139MB/s ± 5%   137MB/s ± 5%     ~     (p=0.204 n=20+20)
CompressTwain-8    532MB/s ± 4%   526MB/s ± 6%     ~     (p=0.283 n=19+20)
CompressRand-8    21.1MB/s ± 4%  21.2MB/s ± 7%     ~     (p=0.635 n=20+20)

(I'm not sure why 1.18 is slower than 1.17 overall.)

It was compressing the wrong file and didn't get enough space to
actually compress its input.
Zeroing out the 128kiB hash table takes a considerable amount of time in
repeated calls to CompressBlock on small, poorly compressible blocks.
Give the hashtable an 8kiB bitmap indicating which entries are in use
and zero that out instead.

Linux/amd64, Go 1.17:

name              old speed      new speed      delta
CompressRandom-8  1.58GB/s ± 2%  1.94GB/s ± 2%  +23.44%  (p=0.000 n=20+19)
CompressPg1661-8   826MB/s ± 5%   837MB/s ± 2%     ~     (p=0.101 n=19+20)
CompressDigits-8   151MB/s ± 1%   150MB/s ± 2%   -0.73%  (p=0.012 n=20+20)
CompressTwain-8    558MB/s ± 2%   559MB/s ± 2%     ~     (p=1.000 n=20+20)
CompressRand-8    25.1MB/s ± 2%  25.1MB/s ± 3%     ~     (p=1.000 n=20+20)

Go 1.18beta:

name              old speed      new speed      delta
CompressRandom-8  1.58GB/s ± 1%  1.95GB/s ± 1%  +23.66%  (p=0.000 n=19+19)
CompressPg1661-8   765MB/s ± 4%   764MB/s ± 6%     ~     (p=0.678 n=20+20)
CompressDigits-8   139MB/s ± 5%   137MB/s ± 5%     ~     (p=0.204 n=20+20)
CompressTwain-8    532MB/s ± 4%   526MB/s ± 6%     ~     (p=0.283 n=19+20)
CompressRand-8    21.1MB/s ± 4%  21.2MB/s ± 7%     ~     (p=0.635 n=20+20)

(I'm not sure why 1.18 is slower than 1.17 overall.)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants