Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: basic criterion benchmarks for store and load of text #77

Merged
merged 1 commit into from
Jan 10, 2021
Merged

perf: basic criterion benchmarks for store and load of text #77

merged 1 commit into from
Jan 10, 2021

Conversation

goller
Copy link
Contributor

@goller goller commented Jan 9, 2021

I had to move src/main.rs into src/bin/fuzzr.rs as benches
cannot import the lib of a binary.

Here is an example report:

Benchmarking load_text_throughput/1024_bytes
Benchmarking load_text_throughput/1024_bytes: Warming up for 3.0000 s
Benchmarking load_text_throughput/1024_bytes: Collecting 100 samples in estimated 5.0522 s (252k iterations)
Benchmarking load_text_throughput/1024_bytes: Analyzing
load_text_throughput/1024_bytes
                        time:   [19.818 us 20.226 us 20.698 us]
                        thrpt:  [47.181 MiB/s 48.282 MiB/s 49.277 MiB/s]
                 change:
                        time:   [-4.0016% -1.3089% +1.4003%] (p = 0.33 > 0.05)
                        thrpt:  [-1.3809% +1.3262% +4.1684%]
                        No change in performance detected.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe
Benchmarking load_text_throughput/16384_bytes
Benchmarking load_text_throughput/16384_bytes: Warming up for 3.0000 s
Benchmarking load_text_throughput/16384_bytes: Collecting 100 samples in estimated 5.0427 s (237k iterations)
Benchmarking load_text_throughput/16384_bytes: Analyzing
load_text_throughput/16384_bytes
                        time:   [21.155 us 21.456 us 21.830 us]
                        thrpt:  [715.76 MiB/s 728.22 MiB/s 738.58 MiB/s]
                 change:
                        time:   [-1.4959% +0.2366% +1.8887%] (p = 0.78 > 0.05)
                        thrpt:  [-1.8537% -0.2360% +1.5186%]
                        No change in performance detected.
Found 14 outliers among 100 measurements (14.00%)
  14 (14.00%) high severe
Benchmarking load_text_throughput/262144_bytes
Benchmarking load_text_throughput/262144_bytes: Warming up for 3.0000 s
Benchmarking load_text_throughput/262144_bytes: Collecting 100 samples in estimated 5.2214 s (91k iterations)
Benchmarking load_text_throughput/262144_bytes: Analyzing
load_text_throughput/262144_bytes
                        time:   [56.666 us 57.265 us 58.031 us]
                        thrpt:  [4.2071 GiB/s 4.2633 GiB/s 4.3085 GiB/s]
                 change:
                        time:   [-1.7947% +0.5167% +3.1932%] (p = 0.71 > 0.05)
                        thrpt:  [-3.0944% -0.5141% +1.8275%]
                        No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) high mild
  9 (9.00%) high severe
Benchmarking load_text_throughput/1024000_bytes
Benchmarking load_text_throughput/1024000_bytes: Warming up for 3.0000 s
Benchmarking load_text_throughput/1024000_bytes: Collecting 100 samples in estimated 5.1192 s (25k iterations)
Benchmarking load_text_throughput/1024000_bytes: Analyzing
load_text_throughput/1024000_bytes
                        time:   [188.78 us 191.93 us 195.37 us]
                        thrpt:  [4.8815 GiB/s 4.9689 GiB/s 5.0519 GiB/s]
                 change:
                        time:   [-2.9584% +0.4243% +4.3536%] (p = 0.82 > 0.05)
                        thrpt:  [-4.1720% -0.4225% +3.0486%]
                        No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe

Benchmarking store_text_throughput/1024_bytes
Benchmarking store_text_throughput/1024_bytes: Warming up for 3.0000 s
Benchmarking store_text_throughput/1024_bytes: Collecting 100 samples in estimated 5.2461 s (66k iterations)
Benchmarking store_text_throughput/1024_bytes: Analyzing
store_text_throughput/1024_bytes
                        time:   [77.428 us 78.358 us 79.522 us]
                        thrpt:  [12.280 MiB/s 12.463 MiB/s 12.613 MiB/s]
                 change:
                        time:   [-3.7098% -1.7624% +0.2276%] (p = 0.08 > 0.05)
                        thrpt:  [-0.2271% +1.7941% +3.8528%]
                        No change in performance detected.
Found 13 outliers among 100 measurements (13.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  11 (11.00%) high severe
Benchmarking store_text_throughput/16384_bytes
Benchmarking store_text_throughput/16384_bytes: Warming up for 3.0000 s
Benchmarking store_text_throughput/16384_bytes: Collecting 100 samples in estimated 5.4411 s (61k iterations)
Benchmarking store_text_throughput/16384_bytes: Analyzing
store_text_throughput/16384_bytes
                        time:   [92.245 us 94.675 us 97.175 us]
                        thrpt:  [160.79 MiB/s 165.04 MiB/s 169.39 MiB/s]
                 change:
                        time:   [+2.0510% +5.4291% +9.5546%] (p = 0.00 < 0.05)
                        thrpt:  [-8.7213% -5.1495% -2.0098%]
                        Performance has regressed.
Found 23 outliers among 100 measurements (23.00%)
  10 (10.00%) high mild
  13 (13.00%) high severe
Benchmarking store_text_throughput/262144_bytes
Benchmarking store_text_throughput/262144_bytes: Warming up for 3.0000 s
Benchmarking store_text_throughput/262144_bytes: Collecting 100 samples in estimated 5.4979 s (20k iterations)
Benchmarking store_text_throughput/262144_bytes: Analyzing
store_text_throughput/262144_bytes
                        time:   [274.98 us 281.16 us 288.82 us]
                        thrpt:  [865.58 MiB/s 889.17 MiB/s 909.15 MiB/s]
                 change:
                        time:   [+3.9246% +6.7403% +9.6361%] (p = 0.00 < 0.05)
                        thrpt:  [-8.7891% -6.3147% -3.7764%]
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild
Benchmarking store_text_throughput/1024000_bytes
Benchmarking store_text_throughput/1024000_bytes: Warming up for 3.0000 s
Benchmarking store_text_throughput/1024000_bytes: Collecting 100 samples in estimated 8.9799 s (10k iterations)
Benchmarking store_text_throughput/1024000_bytes: Analyzing
store_text_throughput/1024000_bytes
                        time:   [831.94 us 853.01 us 877.59 us]
                        thrpt:  [1.0867 GiB/s 1.1180 GiB/s 1.1463 GiB/s]
                 change:
                        time:   [+2.4954% +6.3998% +10.654%] (p = 0.00 < 0.05)
                        thrpt:  [-9.6281% -6.0148% -2.4347%]
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe

I have yet to add image benchmarks; I'm not sure how to generate a variety of sizes yet.

Signed-off-by: Chris Goller [email protected]

@goller
perf: basic criterion benchmarks for store and load
8f03e21 
I had to move src/main.rs into src/bin/fuzzr.rs as benches
cannot import the lib of a binary.

Signed-off-by: Chris Goller <[email protected]>
@cryptoquick
Copy link
Member

Great work also! I'm fine with moving the binary file, and I'll try to make sure everything works well when I do a release. I might have to add a [[bin]] section to the Cargo.toml.

Another issue with forking your changes in, as much as I love the model, is that github doesn't run actions over them. Can you push both of your changes to the repo itself? Also, maybe join our Discord. I should put a link up in the README to our Matrix server, also.

@cryptoquick cryptoquick mentioned this pull request Jan 10, 2021
Merged
@cryptoquick cryptoquick merged commit fc200cb into FuzzrNet:main Jan 10, 2021
@goller goller deleted the perf/load-store-criterion branch January 23, 2021 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@goller @cryptoquick