Code

Device memory spill support#35

Code

Closed

pentschev wants to merge 47 commits intorapidsai:branch-0.8rapidsai/dask-cuda:branch-0.8from pentschev:device-memory-spillpentschev/dask-cuda:device-memory-spillCopy head branch name to clipboard

+863-42

Commits

Commits on Apr 18, 2019

• Add support for yaml configurations

  pentschev

  committed
  1aa68a8

  Copy full SHA for 1aa68a8

• Add DeviceHostFile class to handle memory-spilling in LRU fashion

  pentschev

  committed
  7349974

  Copy full SHA for 7349974

• Add CUDAWorker class to handle device memory

  pentschev

  committed
  377368a

  Copy full SHA for 377368a

• Use CUDAWorker by default, add --device-memory-limit parameter

  pentschev

  committed
  197e653

  Copy full SHA for 197e653

• Rename configurations dask-cuda -> cuda

  pentschev

  committed
  de5c3e9

  Copy full SHA for de5c3e9

Commits on Apr 23, 2019

• Use CUDAWorker on LocalCUDACluster

  pentschev

  committed
  42c0686

  Copy full SHA for 42c0686

• Add scheduler_ip argument to CUDAWorker

  pentschev

  committed
  c92bbc9

  Copy full SHA for c92bbc9

• Add test_device_spill

  pentschev

  committed
  ce4ba37

  Copy full SHA for ce4ba37

• Add assert_device_host_file_size in utils_test module

  pentschev

  committed
  b3cbf2c

  Copy full SHA for b3cbf2c

• Temporarily build with dask-distributed master

  pentschev

  committed
  855876c

  Copy full SHA for 855876c

• Install CuPy for CI builds

  pentschev

  committed
  07fccfa

  Copy full SHA for 07fccfa

Commits on Apr 25, 2019

• Fix DeviceHostFile transfer between host/device dicts

  pentschev

  committed
  6d48805

  Copy full SHA for 6d48805

• Add DeviceHostFile test

  pentschev

  committed
  1dd2c16

  Copy full SHA for 1dd2c16

• Update numpy and numba requirements

  pentschev

  committed
  df6cb95

  Copy full SHA for df6cb95

• Use Dask master and enable __array_function__ for CI builds

  pentschev

  committed
  5a7ceef

  Copy full SHA for 5a7ceef

Commits on Apr 29, 2019

• Fixes for GPU CI

  Show description for 8659fb6
  pentschev

  committed
  8659fb6

  Copy full SHA for 8659fb6

• Fix test_spill, extend it to multiple parameters

  pentschev

  committed
  2c24f03

  Copy full SHA for 2c24f03

• Minor device_host_file and utils_test fixes

  pentschev

  committed
  1d06c75

  Copy full SHA for 1d06c75

• Add CUDANanny, wraps Nanny but defaults to CUDAWorker

  pentschev

  committed
  deff18f

  Copy full SHA for deff18f

Commits on Apr 30, 2019

• Fix GPU indices for test_cuda_visible_devices

  pentschev

  committed
  2d9c150

  Copy full SHA for 2d9c150

• Add single-GPU test for visible devices

  Show description for ce5c650
  pentschev

  committed
  ce5c650

  Copy full SHA for ce5c650

• Improve documentation for CUDANanny, DeviceHostFile

  pentschev

  committed
  c5fbb6f

  Copy full SHA for c5fbb6f

Commits on May 1, 2019

• Fix defaults for device_memory_limit

  pentschev

  committed
  dcc6a6a

  Copy full SHA for dcc6a6a

• Fix test_with_subset_of_cuda_visible_devices

  pentschev

  committed
  9bc21e7

  Copy full SHA for 9bc21e7

Commits on May 2, 2019

• Pass *args to CUDANanny and CUDAWorker

  pentschev

  committed
  9eb2dfd

  Copy full SHA for 9eb2dfd

• Increase required dask, distributed versions

  pentschev

  committed
  4324439

  Copy full SHA for 4324439

• Use yaml.safe_load

  mrocklin

  committed
  388c677

  Copy full SHA for 388c677

Commits on May 3, 2019

• Fix CUDA worker not pausing issue, print correct resume message

  pentschev

  committed
  c942438

  Copy full SHA for c942438

• Support protocol= keyword in LocalCUDACluster (rapidsai#39)

  Show description for d968b0f
  mrocklin

  authored
  d968b0f

  Copy full SHA for d968b0f

Commits on May 6, 2019

• Merge branch 'branch-0.7' into device-memory-spill

  mrocklin

  authored
  1abb1eb

  Copy full SHA for 1abb1eb

• Merge branch 'device-memory-spill' of github.com:pentschev/dask-cuda into device-memory-spill

  mrocklin

  committed
  eb70191

  Copy full SHA for eb70191

• Minor style changes

  mrocklin

  committed
  806ac8b

  Copy full SHA for 806ac8b

• Change config namespace from cuda to distributed

  mrocklin

  committed
  f35826e

  Copy full SHA for f35826e

• Rename files to remove cuda suffix

  Show description for f308943
  mrocklin

  committed
  f308943

  Copy full SHA for f308943

• Fix style

  pentschev

  committed
  1b9d38b

  Copy full SHA for 1b9d38b

Commits on May 7, 2019

• Add device, host, disk LRU aliases to simplify user queries

  pentschev

  committed
  6d5b714

  Copy full SHA for 6d5b714

• Default pausing due to device memory usage to 0.0 (disable)

  pentschev

  committed
  358f194

  Copy full SHA for 358f194

• Small doc fix in dask_cuda_worker.py

  Show description for c81f763
  

  mrocklin

  and

  pentschev

  authored
  c81f763

  Copy full SHA for c81f763

• Avoid modifying DeviceHostFile internals in test_device_host_file

  pentschev

  committed
  f28cbd1

  Copy full SHA for f28cbd1

• Merge remote-tracking branch 'origin/device-memory-spill' into device-memory-spill

  pentschev

  committed
  6163359

  Copy full SHA for 6163359

• Allow choosing worker-class when starting dask-cuda-worker

  pentschev

  committed
  a3c89fb

  Copy full SHA for a3c89fb

• Add back original test_cuda_visible_devices, using Worker class

  pentschev

  committed
  8f59141

  Copy full SHA for 8f59141

• Aliasing device, host and disk to dict instead of LRU

  pentschev

  committed
  1e3acce

  Copy full SHA for 1e3acce

• Remove utils_test, use simple naming instead of gen_random_key()

  pentschev

  committed
  882bf31

  Copy full SHA for 882bf31

• Use tmp_path instead of tempfile, fix flake8/black issues

  pentschev

  committed
  1960b79

  Copy full SHA for 1960b79

• Add back 'loop' to test_cuda_visible_devices, despite flake8 complaints

  pentschev

  committed
  eb86d5d

  Copy full SHA for eb86d5d

• Add missing pause argument to test_spill

  pentschev

  committed
  8153b27

  Copy full SHA for 8153b27