Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] dask-cuda v23.04 #1158

Merged
merged 35 commits into from
Apr 12, 2023
Merged

[RELEASE] dask-cuda v23.04 #1158

merged 35 commits into from
Apr 12, 2023

Conversation

ajschmidt8
Copy link
Member

❄️ Code freeze for branch-23.04 and v23.04 release

What does this mean?

Only critical/hotfix level issues should be merged into branch-23.04 until release (merging of this PR).

What is the purpose of this PR?

  • Update documentation
  • Allow testing for the new release
  • Enable a means to merge branch-23.04 into main for the release

raydouglass and others added 30 commits January 23, 2023 10:50
@raydouglass
DOC
074cf16
@GPUtester
Merge pull request #1093 from rapidsai/branch-23.02
b534e84 
Forward-merge branch-23.02 to branch-23.04
@GPUtester
Merge pull request #1095 from rapidsai/branch-23.02
221a810 
Forward-merge branch-23.02 to branch-23.04
@GPUtester
Merge pull request #1097 from rapidsai/branch-23.02
3796bb8 
Forward-merge branch-23.02 to branch-23.04
@GPUtester
Merge pull request #1099 from rapidsai/branch-23.02
db36677 
Forward-merge branch-23.02 to branch-23.04
@GPUtester
Merge pull request #1100 from rapidsai/branch-23.02
410d814 
Forward-merge branch-23.02 to branch-23.04
@GPUtester
Merge pull request #1102 from rapidsai/branch-23.02
65a02b1 
Forward-merge branch-23.02 to branch-23.04
@GPUtester
Merge pull request #1107 from rapidsai/branch-23.02
d0ca59d 
Forward-merge branch-23.02 to branch-23.04
@GPUtester
Merge pull request #1108 from rapidsai/branch-23.02
7c5d0b9 
Forward-merge branch-23.02 to branch-23.04
@ajschmidt8
Move date to build string in conda recipe (#1103)
014d588 
This PR moves the date string from the version to the build string for conda recipes in this repository.

This is necessary to ensure that the conda packages resulting from PR builds can be installed in the same environment as nightly conda packages, which is useful for testing purposes.

Additionally, this prevents a bug from occurring where the Python builds fail because the date string it computes is different than the one computed by the C++ build, therefore causing the Python build to search for a C++ build artifact that doesn't exist.

xref: rapidsai/rmm#1195

Authors:
  - AJ Schmidt (https://github.com/ajschmidt8)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)

URL: #1103
@ajschmidt8
Update shared workflow branches (#1112)
4f4aac1 
This PR updates the branch reference used for our shared workflows.

Authors:
  - AJ Schmidt (https://github.com/ajschmidt8)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)

URL: #1112
@GPUtester
Merge pull request #1114 from rapidsai/branch-23.02
7d4d1c7 
Forward-merge branch-23.02 to branch-23.04
@galipremsagar
Unpin dask and distributed for development (#1110)
b0e4f2d 
This PR unpins `dask` and `distributed` for `23.04` development.


xref: rapidsai/cudf#12710

Authors:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Ray Douglass (https://github.com/raydouglass)

URL: #1110
@AjayThorve
Reduce error handling verbosity in CI tests scripts (#1113)
f44ba33 
This PR adds a less verbose [trap method](https://github.com/rapidsai/cugraph/blob/f2b081075704aabc789603e14ce552eac3fbe692/ci/test.sh#L19), for error handling to help ensure that we capture all potential error codes in our test scripts, and works as follows:

- setting an environment variable, EXITCODE, with a default value of 0
- setting a trap statement triggered by ERR signals which will set EXITCODE=1 when any commands return a non-zero exit code
cc @ajschmidt8

Authors:
  - Ajay Thorve (https://github.com/AjayThorve)

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: #1113
@bdice
Remove gpuCI scripts. (#1111)
4d0bdfd 
Since migrating to GitHub Actions, the gpuCI scripts are no longer needed. This PR removes those outdated gpuCI scripts.

Authors:
  - Bradley Dice (https://github.com/bdice)
  - AJ Schmidt (https://github.com/ajschmidt8)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)
  - Mads R. B. Kristensen (https://github.com/madsbk)
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: #1111
@pentschev
Allow using stream-ordered and managed RMM allocators in benchmarks (#…
a0fda76 
…1012)

This change requires UCX 1.14 for async support from openucx/ucx#8623 and UVM fixes from openucx/ucx#8754 .

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Benjamin Zaitlen (https://github.com/quasiben)

URL: #1012
@pentschev
Fix for bytes/str discrepancy after PyNVML update (#1118)
8134e6b 
A PyNVML update has changed how some objects were previously returned bytes but now return str, which is now handled appropriately with this change.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #1118
@ajschmidt8
Merge branch-23.02 into merge-23.02-into-23.04
461ff29
@ajschmidt8
Merge pull request #1128 from ajschmidt8/merge-23.02-into-23.04
95461ea 
Merge `branch-23.02` into `branch-23.04`
@AjayThorve
Fix GHA build workflow (#1120)
458655d 
This PR fixes the condition to trigger docs-build workflow in build.yaml to:

- simplify the branch assertion
- ensure that it only runs on push events (as opposed to workflow_dispatch events which trigger the nightlies).

@ajschmidt8

Authors:
  - Ajay Thorve (https://github.com/AjayThorve)
  - AJ Schmidt (https://github.com/ajschmidt8)

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: #1120
@jakirkham
Template Conda recipe's about metadata (#1121)
af0f3ef 
Pull in the Conda recipe's `about` metadata from `pyproject.toml` using templating. Follows more closely with DRY and move more towards using `pyproject.toml` as the single source of truth.

Authors:
  - https://github.com/jakirkham

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)
  - Peter Andreas Entschev (https://github.com/pentschev)

URL: #1121
@madsbk
Explicit-comms: update monkey patching of Dask (#1135)
92190af 
Closes #1134 by wrapping `rearrange_by_column()` instead of `rearrange_by_column_tasks()`.  

This also has the bonus that we avoid a re-partition when the shuffle changes number of partitions: https://github.com/dask/dask/blob/945f4e8b7646228aff34da07ffaa52f1b73aa1e0/dask/dataframe/shuffle.py#L510

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)

URL: #1135
@jakirkham
Drop Python 3.7 handling for pickle protocol 4 (#1132)
7c0bde1 
Fixes #1131

Now that Python 3.8 is the minimum supported version, drop the special casing for Python 3.7's `HIGHEST_PROTOCOL`, which was 4 (not 5). In Python 3.8+, `HIGHEST_PROTOCOL >= 5`. So none of these branches are needed any more.

Authors:
  - https://github.com/jakirkham
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Mads R. B. Kristensen (https://github.com/madsbk)
  - Peter Andreas Entschev (https://github.com/pentschev)

URL: #1132
@wence-
Adapt to rapidsai/rmm#1221 which moves allocator callbacks (#1129)
b9561cf 
The allocator callbacks now live in their own submodules (so that RMM does not, for example, import pytorch unless required) and so must be explicitly imported.

Authors:
  - Lawrence Mitchell (https://github.com/wence-)
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)

URL: #1129
@pentschev
Extend RMM async allocation support (#1116)
3fadc30 
Ensure pool size argument is respected when enabling RMM async allocator, add release threshold support.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #1116
@madsbk
Serialize of ProxyObject to pickle fixed attributes (#1137)
64a92a1 
The proxied's `name` attribute might contain types not support by msgpack. We now pickle the fixed attributes when serializing. 

Closes #1136

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - Peter Andreas Entschev (https://github.com/pentschev)

URL: #1137
@jakirkham
Update rmm_cupy_allocator usage (#1138)
1a35569 
Follow up to PR ( #1129 ) and PR ( rapidsai/rmm#1221 )

Uses `rmm_cupy_allocator` from `rmm.allocators.cupy` where it has been moved to recently.

cc @wence-

Authors:
  - https://github.com/jakirkham

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)

URL: #1138
@galipremsagar
Update minimum pandas and numpy pinnings (#1139)
cdd4c0b 
This PR updates `pandas` & `numpy` pinnings to be in-sync with `cudf`: rapidsai/cudf#12887

Authors:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Ray Douglass (https://github.com/raydouglass)

URL: #1139
@charlesbluca
Reinstate --death-timeout CLI option (#1140)
ec2de78 
Add back in the `--death-timeout` option removed in #563, along with some tests to verify it's working as expected.

Closes #1017

Authors:
  - Charles Blackmon-Luca (https://github.com/charlesbluca)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)

URL: #1140
@pentschev
Update usage of get_worker() in tests (#1141)
06fb4e2 
In dask/distributed#7580 `get_worker` was modified to return the worker of a task, thus it cannot be used by `client.run`, and we must now use `dask_worker` as the first argument to `client.run` to obtain the worker.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Mads R. B. Kristensen (https://github.com/madsbk)
  - Lawrence Mitchell (https://github.com/wence-)

URL: #1141
madsbk and others added 5 commits March 28, 2023 06:02
@madsbk
Monkey patching all locations of get_default_shuffle_algorithm (#1142)
9fef6b7 
Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)

URL: #1142
@pentschev
Add argument to enable RMM alloaction tracking in benchmarks (#1145)
2079152 
Tracking RMM allocation will be useful together with dask/distributed#5740 , and will help with the analysis of memory fragmentation when comparing regular pool and the async memory allocator.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Benjamin Zaitlen (https://github.com/quasiben)

URL: #1145
@pentschev
Allow specifying dashboard address in benchmarks (#1147)
a51b10b 
This is useful for shared machines where the user may not have control of the default port `8787`.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - Jacob Tomlinson (https://github.com/jacobtomlinson)

URL: #1147
@galipremsagar
Pin dask and distributed for release (#1153)
724e1c6 
This PR pins `dask` and `distributed` to `2023.3.2` and `2023.3.2.1` respectively for `23.04` release.

xref: rapidsai/cudf#13070

Authors:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Ray Douglass (https://github.com/raydouglass)

URL: #1153
@galipremsagar
Rectify dask-core pinning in pip requirements (#1155)
1f51c4f 
As part of #1153 `dask-core` has been added to pip requirements, which is incorrect. This PR rectifies this issue.

Authors:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)

URL: #1155
@ajschmidt8 ajschmidt8 requested review from a team as code owners April 7, 2023 14:24
@github-actions github-actions bot added conda conda issue gpuCI gpuCI issue python python code needed labels Apr 7, 2023
@raydouglass raydouglass merged commit a301937 into main Apr 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
conda conda issue gpuCI gpuCI issue python python code needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.