Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: take_run benchmark parameter #3679

Merged
merged 1 commit into from
Feb 9, 2023
Merged

fix: take_run benchmark parameter #3679

merged 1 commit into from
Feb 9, 2023

Conversation

askoa
Copy link
Contributor

@askoa askoa commented Feb 8, 2023

Which issue does this PR close?

Rationale for this change

I recently noticed an issue with the parameters in the benchmark primitive_run_take. The benchmark was not taking indices across the entire logical array but rather constrained to the length of take array. I updated the parameters to take indices across the entire logical array. There is performance regression because more values are taken from the array. I have furnished the performance differences below

Benchmark results

primitive_run_take/(run_array_len:512, physical_array_len:64, take_len:512)
                        time:   [23.347 µs 23.508 µs 23.713 µs]
                        change: [-1.5238% +0.0502% +1.6099%] (p = 0.95 > 0.05)
                        No change in performance detected.
Found 17 outliers among 100 measurements (17.00%)
  3 (3.00%) high mild
  14 (14.00%) high severe
primitive_run_take/(run_array_len:512, physical_array_len:128, take_len:512)
                        time:   [23.477 µs 23.616 µs 23.804 µs]
                        change: [-1.2725% +0.2827% +1.9661%] (p = 0.75 > 0.05)
                        No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) high mild
  10 (10.00%) high severe
primitive_run_take/(run_array_len:1024, physical_array_len:256, take_len:512)
                        time:   [23.653 µs 23.778 µs 23.975 µs]
                        change: [-1.3315% +0.3299% +2.0635%] (p = 0.72 > 0.05)
                        No change in performance detected.
Found 13 outliers among 100 measurements (13.00%)
  3 (3.00%) high mild
  10 (10.00%) high severe
primitive_run_take/(run_array_len:1024, physical_array_len:256, take_len:1024)
                        time:   [47.861 µs 48.179 µs 48.621 µs]
                        change: [-0.3026% +1.4710% +3.2931%] (p = 0.10 > 0.05)
                        No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe
primitive_run_take/(run_array_len:2048, physical_array_len:512, take_len:512)
                        time:   [24.226 µs 24.287 µs 24.356 µs]
                        change: [+0.7574% +3.6336% +7.6136%] (p = 0.02 < 0.05)
                        Change within noise threshold.
Found 11 outliers among 100 measurements (11.00%)
  3 (3.00%) high mild
  8 (8.00%) high severe
primitive_run_take/(run_array_len:2048, physical_array_len:512, take_len:1024)
                        time:   [53.774 µs 54.093 µs 54.615 µs]
                        change: [+12.948% +15.613% +18.678%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
  2 (2.00%) high mild
  11 (11.00%) high severe
primitive_run_take/(run_array_len:4096, physical_array_len:1024, take_len:512)
                        time:   [25.297 µs 25.365 µs 25.440 µs]
                        change: [+4.7053% +7.0111% +9.3052%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
  5 (5.00%) high mild
  8 (8.00%) high severe
primitive_run_take/(run_array_len:4096, physical_array_len:1024, take_len:1024)
                        time:   [59.299 µs 59.752 µs 60.351 µs]
                        change: [+21.297% +23.389% +25.448%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 16 outliers among 100 measurements (16.00%)
  8 (8.00%) high mild
  8 (8.00%) high severe

What changes are included in this PR?

Are there any user-facing changes?

@github-actions github-actions bot added the arrow Changes to the arrow crate label Feb 8, 2023
@tustvold tustvold merged commit 98ce68f into apache:master Feb 9, 2023
@ursabot
Copy link

ursabot commented Feb 9, 2023

Benchmark runs are scheduled for baseline = 7cd29d7 and contender = 98ce68f. 98ce68f is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@askoa askoa deleted the take-run-bm branch February 9, 2023 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants