[REVIEW] Fix device memory spilling with cuDF #65

pentschev · 2019-05-31T19:00:45Z

Solves #57

mrocklin · 2019-05-31T20:18:55Z

mrocklin · 2019-05-31T20:20:06Z

dask_cuda/device_host_file.py

+    except ImportError:
+        _device_instances = []
+    return (hasattr(obj, "__cuda_array_interface__") or
+            any([isinstance(obj, inst) for inst in _device_instances]))


Importing cudf is non-trivially expensive (a few seconds). Also, it would be nice not to have to special case libraries like this. I wonder if there is another signal we can use in cases like these. cc @kkraus14

I do think though, that testing that cudf objects register as device objects would be very helpful. Same with lists of cudf objects (and other objects like cupy arrays and numba device arrays).

Thanks for the reminder. I agree that we would be better off not testing specifically for cudf objects, but currently there's no clean way of doing it. I've commented on this before as well #65 (comment). For now, I'm checking those with __module__ to avoid importing cudf.

Once we extend __cuda_array_interface__ with the mask attribute we'll be able to handle cudf.Series nicely, but cudf.DataFrame will still need to be special cased.

Given we'd need to special case any non-array-like GPU objects for this, i.e. an XGBoost DMatrix backed by device memory, it would be good if we could design a generalized approach.

A generalized approach would be great, but AFAIK, there's no error-free way to check whether an object is backed by device memory or not. Do you know of any such way @kkraus14?

He might be suggesting something like a type registry or dispatch function, maybe similar to how we handle the sizeof function today.

This is a good idea. I've added that in the latest commit.

mrocklin · 2019-05-31T20:20:40Z

dask_cuda/device_host_file.py

+    if isinstance(obj, list) or isinstance(obj, tuple):
+        return any([_is_device_object(o) for o in obj])
+    else:
+        return _is_device_object(obj)


Maybe roll this logic into _is_device_object directly?

Since _is_device_object is not just a single condition now, I'd rather keep it separate than duplicating that code.

Perhaps some light recursion:

def is_device_object(obj): if isinstance(obj, (list, tuple)): return any(map(is_device_object, obj)) elif ...

pentschev · 2019-05-31T20:34:12Z

Importing cudf is non-trivially expensive (a few seconds). Also, it would be nice not to have to special case libraries like this. I wonder if there is another signal we can use in cases like these.

Yeah, I actually overlooked that, thanks for pointing it out. I checked on side channels whether we have a good way of doing this, but we currently don't. What we can check is for the existence of the as_gpu_matrix function via hasattr, but this isn't guaranteed to be exclusive to cuDF, and that's why I tried something more certain.

pentschev · 2019-05-31T20:34:45Z

I do think though, that testing that cudf objects register as device objects would be very helpful. Same with lists of cudf objects (and other objects like cupy arrays and numba device arrays).

I agree, I'm working on a test, I forgot to mark this as [WIP], doing it now.

VibhuJawa · 2019-06-01T00:57:31Z

So, I tried this with a slightly bigger example and I am getting the below error in the worker logs. The computation seems to be paused.

distributed.worker - ERROR - [1] Call to cuMemcpyHtoD results in CUDA_ERROR_INVALID_VALUE Traceback (most recent call last): File "/conda/envs/rapids/lib/python3.7/site-packages/distributed/worker.py", line 2037, in release_key if key in self.data and key not in self.dep_state: File "/conda/envs/rapids/lib/python3.7/_collections_abc.py", line 666, in __contains__ self[key] File "/conda/envs/rapids/lib/python3.7/site-packages/dask_cuda-0.0.0.dev0-py3.7.egg/dask_cuda/device_host_file.py", line 111, in __getitem__ self.device_buffer[key] = _deserialize_if_device(obj) File "/conda/envs/rapids/lib/python3.7/site-packages/dask_cuda-0.0.0.dev0-py3.7.egg/dask_cuda/device_host_file.py", line 47, in _deserialize_if_device return deserialize_bytes(obj) File "/conda/envs/rapids/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 392, in deserialize_bytes return deserialize(header, frames) File "/conda/envs/rapids/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 190, in deserialize return loads(header, frames) File "/conda/envs/rapids/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 64, in pickle_loads return pickle.loads(b"".join(frames)) File "/conda/envs/rapids/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 61, in loads return pickle.loads(x) File "/conda/envs/rapids/lib/python3.7/site-packages/cudf-0.8.0a1+348.g2a7237c.dirty-py3.7-linux-x86_64.egg/cudf/dataframe/buffer.py", line 40, in __init__ self.mem = cudautils.to_device(mem) File "/conda/envs/rapids/lib/python3.7/site-packages/cudf-0.8.0a1+348.g2a7237c.dirty-py3.7-linux-x86_64.egg/cudf/utils/cudautils.py", line 22, in to_device dary, _ = rmm.auto_device(ary) File "/conda/envs/rapids/lib/python3.7/site-packages/librmm_cffi-0.8.0-py3.7.egg/librmm_cffi/wrapper.py", line 268, in auto_device devobj.copy_to_device(obj, stream=stream) File "/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/devices.py", line 212, in _require_cuda_context return fn(*args, **kws) File "/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/devicearray.py", line 198, in copy_to_device _driver.host_to_device(self, ary_core, self.alloc_size, stream=stream) File "/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 1838, in host_to_device fn(device_pointer(dst), host_pointer(src, readonly=True), size, *varargs) File "/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 293, in safe_cuda_api_call self._check_error(fname, retcode) File "/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 328, in _check_error raise CudaAPIError(retcode, msg) numba.cuda.cudadrv.driver.CudaAPIError: [1] Call to cuMemcpyHtoD results in CUDA_ERROR_INVALID_VALUE

distributed.worker - ERROR - [1] Call to cuMemcpyHtoD results in CUDA_ERROR_INVALID_VALUE Traceback (most recent call last): File "/conda/envs/rapids/lib/python3.7/site-packages/distributed/worker.py", line 2037, in release_key if key in self.data and key not in self.dep_state: File "/conda/envs/rapids/lib/python3.7/_collections_abc.py", line 666, in __contains__ self[key] File "/conda/envs/rapids/lib/python3.7/site-packages/dask_cuda-0.0.0.dev0-py3.7.egg/dask_cuda/device_host_file.py", line 111, in __getitem__ self.device_buffer[key] = _deserialize_if_device(obj) File "/conda/envs/rapids/lib/python3.7/site-packages/dask_cuda-0.0.0.dev0-py3.7.egg/dask_cuda/device_host_file.py", line 47, in _deserialize_if_device return deserialize_bytes(obj) File "/conda/envs/rapids/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 392, in deserialize_bytes return deserialize(header, frames) File "/conda/envs/rapids/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 190, in deserialize return loads(header, frames) File "/conda/envs/rapids/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 64, in pickle_loads return pickle.loads(b"".join(frames)) File "/conda/envs/rapids/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 61, in loads return pickle.loads(x) File "/conda/envs/rapids/lib/python3.7/site-packages/cudf-0.8.0a1+348.g2a7237c.dirty-py3.7-linux-x86_64.egg/cudf/dataframe/buffer.py", line 40, in __init__ self.mem = cudautils.to_device(mem) File "/conda/envs/rapids/lib/python3.7/site-packages/cudf-0.8.0a1+348.g2a7237c.dirty-py3.7-linux-x86_64.egg/cudf/utils/cudautils.py", line 22, in to_device dary, _ = rmm.auto_device(ary) File "/conda/envs/rapids/lib/python3.7/site-packages/librmm_cffi-0.8.0-py3.7.egg/librmm_cffi/wrapper.py", line 268, in auto_device devobj.copy_to_device(obj, stream=stream) File "/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/devices.py", line 212, in _require_cuda_context return fn(*args, **kws) File "/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/devicearray.py", line 198, in copy_to_device _driver.host_to_device(self, ary_core, self.alloc_size, stream=stream) File "/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 1838, in host_to_device fn(device_pointer(dst), host_pointer(src, readonly=True), size, *varargs) File "/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 293, in safe_cuda_api_call self._check_error(fname, retcode) File "/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 328, in _check_error raise CudaAPIError(retcode, msg) numba.cuda.cudadrv.driver.CudaAPIError: [1] Call to cuMemcpyHtoD results in CUDA_ERROR_INVALID_VALUE

distributed.worker - ERROR - '_compare_frame-b40800fb-3765-4e87-a046-9f64170b5039' Traceback (most recent call last): File "/conda/envs/rapids/lib/python3.7/site-packages/distributed/worker.py", line 2123, in release_dep if self.task_state[key] != "memory": KeyError: '_compare_frame-b40800fb-3765-4e87-a046-9f64170b5039'

distributed.worker - ERROR - Key not ready to send to worker, _compare_frame-b40800fb-3765-4e87-a046-9f64170b5039: memory

distributed.worker - ERROR - Key not ready to send to worker, _compare_frame-da2444cf-fa27-45da-9aea-2f3851393073: memory

Can you, @pentschev confirm that it worked in the example in #57 ?

pentschev · 2019-06-01T12:35:16Z

@VibhuJawa yes, your example was my debugging and test case, and it works now with the changes in this PR. I've also tested with more devices, a few different device_memory_limit values, and both RAPIDS 0.7 and nightly, and all of them worked for me. Could you try running that on your side as well and confirming if that works?

VibhuJawa · 2019-06-03T20:52:03Z

@VibhuJawa yes, your example was my debugging and test case, and it works now with the changes in this PR. I've also tested with more devices, a few different device_memory_limit values, and both RAPIDS 0.7 and nightly, and all of them worked for me. Could you try running that on your side as well and confirming if that works?

@pentschev . Can confirm, The posted example works.

Unrelated, I am getting the below error when i set the memory_limit (host).

from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import cudf, dask_cudf

# Use dask-cuda to start one worker per GPU on a single-node system
# When you shutdown this notebook kernel, the Dask cluster also shuts down.
cluster = LocalCUDACluster(ip='0.0.0.0',n_workers=1, device_memory_limit='10000 MiB',memory_limit='16000 MiB')
client = Client(cluster)
# # print client info
print(client)

# Code to simulate_data

def generate_file(output_file,rows=100):
    with open(output_file, 'wb') as f:
        f.write(b'A,B,C,D,E,F,G,H,I,J,K\n')
        f.write(b'22,697,56,0.0,0.0,0.0,0.0,0.0,0.0,0,0\n23,697,56,0.0,0.0,0.0,0.0,0.0,0.0,0,0\n'*(rows//2))
        f.close()

# generate the test file 
output_file='test.csv'

# reading it using dask_cudf
df = dask_cudf.read_csv(output_file,chunksize='100 MiB')
print(len(df))

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-7c819d29785c> in <module>
     23 # reading it using dask_cudf
     24 df = dask_cudf.read_csv(output_file,chunksize='100 MiB')
---> 25 print(len(df))

/conda/envs/rapids/lib/python3.7/site-packages/dask/dataframe/core.py in __len__(self)
    455     def __len__(self):
    456         return self.reduction(len, np.sum, token='len', meta=int,
--> 457                               split_every=False).compute()
    458 
    459     def __bool__(self):

/conda/envs/rapids/lib/python3.7/site-packages/dask/base.py in compute(self, **kwargs)
    154         dask.base.compute
    155         """
--> 156         (result,) = compute(self, traverse=False, **kwargs)
    157         return result
    158 

/conda/envs/rapids/lib/python3.7/site-packages/dask/base.py in compute(*args, **kwargs)
    396     keys = [x.__dask_keys__() for x in collections]
    397     postcomputes = [x.__dask_postcompute__() for x in collections]
--> 398     results = schedule(dsk, keys, **kwargs)
    399     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    400 

/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   2566                     should_rejoin = False
   2567             try:
-> 2568                 results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   2569             finally:
   2570                 for f in futures.values():

/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py in gather(self, futures, errors, maxsize, direct, asynchronous)
   1820                 direct=direct,
   1821                 local_worker=local_worker,
-> 1822                 asynchronous=asynchronous,
   1823             )
   1824 

/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py in sync(self, func, *args, **kwargs)
    751             return future
    752         else:
--> 753             return sync(self.loop, func, *args, **kwargs)
    754 
    755     def __repr__(self):

/conda/envs/rapids/lib/python3.7/site-packages/distributed/utils.py in sync(loop, func, *args, **kwargs)
    329             e.wait(10)
    330     if error[0]:
--> 331         six.reraise(*error[0])
    332     else:
    333         return result[0]

/conda/envs/rapids/lib/python3.7/site-packages/six.py in reraise(tp, value, tb)
    691             if value.__traceback__ is not tb:
    692                 raise value.with_traceback(tb)
--> 693             raise value
    694         finally:
    695             value = None

/conda/envs/rapids/lib/python3.7/site-packages/distributed/utils.py in f()
    314             if timeout is not None:
    315                 future = gen.with_timeout(timedelta(seconds=timeout), future)
--> 316             result[0] = yield future
    317         except Exception as exc:
    318             error[0] = sys.exc_info()

/conda/envs/rapids/lib/python3.7/site-packages/tornado/gen.py in run(self)
    727 
    728                     try:
--> 729                         value = future.result()
    730                     except Exception:
    731                         exc_info = sys.exc_info()

/conda/envs/rapids/lib/python3.7/site-packages/tornado/gen.py in run(self)
    734                     if exc_info is not None:
    735                         try:
--> 736                             yielded = self.gen.throw(*exc_info)  # type: ignore
    737                         finally:
    738                             # Break up a reference to itself

/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
   1651                             six.reraise(CancelledError, CancelledError(key), None)
   1652                         else:
-> 1653                             six.reraise(type(exception), exception, traceback)
   1654                     if errors == "skip":
   1655                         bad_keys.add(key)

/conda/envs/rapids/lib/python3.7/site-packages/six.py in reraise(tp, value, tb)
    690                 value = tp()
    691             if value.__traceback__ is not tb:
--> 692                 raise value.with_traceback(tb)
    693             raise value
    694         finally:

/conda/envs/rapids/lib/python3.7/site-packages/dask_cuda-0.0.0.dev0-py3.7.egg/dask_cuda/device_host_file.py in __setitem__()

/conda/envs/rapids/lib/python3.7/site-packages/zict/buffer.py in __setitem__()
     75         weight = self.weight(key, value)
     76         # Avoid useless movement for heavy values
---> 77         if self.weight(key, value) <= self.n:
     78             if key in self.slow:
     79                 del self.slow[key]

TypeError: '<=' not supported between instances of 'int' and 'str'

</details>

Edit: Put long trace in details.

pentschev · 2019-06-03T21:23:19Z

@VibhuJawa thanks for reporting that. Certainly something small with the parsing/storing memory_limit. In the meantime, if you need to continue with your tests, you can use device_memory=int(16e9) (which you probably know already, my apologies if you do).

VibhuJawa · 2019-06-03T21:36:15Z

Thanks for that.

I think the previous error (#65 (comment)) was due to a large chunk-size i was using. Will try to create a minimal example that i can post. (I need larger chunksizes to make sorting more efficient)

The previous run was on private data so cant share that here.

beckernick · 2019-06-04T19:14:34Z

cc @galipremsagar for visibility as well

…ce-memory-spill

dask_cuda/tests/test_spill.py

pentschev · 2019-06-11T12:12:43Z

This number for host memory limit is quite low. Just importing a few libraries like Pandas and a few other libraries can get us close to this point.

Yes, I was trying to shorten test time by running on smaller data, but I think that's not really useful if it doesn't work, due to such small limits. Maybe increasing that alone will fix the issues.

pentschev · 2019-06-12T20:04:51Z

@mrocklin I managed to tackle the timeout/memory limit issues that were causing failures. This is now good for a review.

mrocklin · 2019-06-13T07:19:53Z

dask_cuda/tests/test_spill.py

    @gen_cluster(
        client=True,
        ncores=[("127.0.0.1", 1)],
        Worker=Worker,
+        timeout=300,


How long does this test usually take? Five minutes seems like a very long time. Is this timeout still necessary?

I think most of the tests here run in under 60 seconds, but the default isn't enough for some of them, in particular there was one of the CuPy tests that I've seen failing non-deterministically. Since we're not concerned about the timeout in itself for the tests here, IMHO, it's better to have longer timeouts than having some tests failing now and then. In other words, the 300 seconds becomes more of a trigger to avoid CI from hanging forever in an eventual failure, but also prevents it from failing when it just took a while longer due to some unexpected slowness of the CI system.

mrocklin · 2019-06-13T07:22:52Z

dask_cuda/tests/test_spill.py

+        yield client.run(worker_assert, nbytes, 32, 2048 + part_index_nbytes)
+
+        host_chunks = yield client.run(lambda: len(get_worker().data.host))
+        disk_chunks = yield client.run(lambda: len(get_worker().data.disk))


It's not important, but because you're using normal Workers here rather than Nannys, have direct access to the workers here. You can look at worker.data.host and worker.data.disk directly, without using client.run.

Yes, but I prefer to call client.run here to make sure that works and to make all tests consistent, I've added some more client.run now. Perhaps we can generalize that in the future to a single assertion function.

mrocklin · 2019-06-13T07:23:32Z

dask_cuda/device_host_file.py

+    if isinstance(obj, list) or isinstance(obj, tuple):
+        return any([_is_device_object(o) for o in obj])
+    else:
+        return _is_device_object(obj)


mrocklin · 2019-06-13T07:23:38Z

dask_cuda/device_host_file.py

+    except ImportError:
+        _device_instances = []
+    return (hasattr(obj, "__cuda_array_interface__") or
+            any([isinstance(obj, inst) for inst in _device_instances]))


galipremsagar · 2019-06-14T00:09:44Z

@pentschev

This PR does work in case of the snippet which @VibhuJawa posted in this link: #57 (comment)

But below code is causing an out of memory exception and not controlling spilling over of memory.

from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import cudf, dask_cudf

cluster = LocalCUDACluster(ip='0.0.0.0',n_workers=1, device_memory_limit='10000 MiB')
client = Client(cluster)

print(client)


# Code to simulate_data

def generate_file(output_file,rows=100):
    with open(output_file, 'wb') as f:
        f.write(b'A,B,C,D,E,F,G,H,I,J,K\n')
        f.write(b'127.0.0.1,697,56,0.0,0.0,0.0,0.0,0.0,0.0,0,0\n121.1.2.4,697,56,0.0,0.0,0.0,0.0,0.0,0.0,0,0\n'*(rows//2))
        f.close()

# generate the test file 


output_file='test.csv'
# Uncomment below
generate_file(output_file,rows=100_000_000_0)

# reading it using dask_cudf


df = dask_cudf.read_csv(output_file,chunksize='100 MiB')
print(df.head(10).to_pandas())


# converting all IPs to integer(long) values
def to_long(df):
    gpu_strings = df['A'].data
    df['int_ips'] = gpu_strings.ip2int()
    return df


long_df = df.map_partitions(to_long)
x = long_df.persist()
wait(x)

pentschev · 2019-06-14T07:09:07Z

@galipremsagar thanks for reminding me of that case. This is a discussion @VibhuJawa and I had on side channels, I reposted it in #57 (comment) for visibility.

pentschev · 2019-06-20T21:48:11Z

For the case @galipremsagar asked about, unfortunately, this can't be handled from the dask-cuda side alone, and will likely require changes to cuDF in the future. The only solution for the time being is to reduce chunksize.

pentschev · 2019-06-20T21:49:45Z

Regardless of the issues with specific pipelines, this PR solves the general cuDF cases where the memory is exposed on the Python side. Therefore, @mrocklin please review so we can get it merged before the code freeze tomorrow!

ci/gpu/build.sh

dask_cuda/is_device_object.py

Co-Authored-By: Keith Kraus <[email protected]>

pentschev · 2019-06-21T08:06:38Z

Thanks @kkraus14 for the review!

…ce-memory-spill

…to fix-cudf-device-memory-spill

mrocklin · 2019-06-21T10:49:24Z

This seems fine to me

pentschev · 2019-06-21T12:13:20Z

Thanks for the review @mrocklin, merging.

pentschev added 2 commits May 31, 2019 10:04

Checking device objects include cuDF DataFrame and Series

b8725bb

Check if any of tuple/list objects are device objects

fea3133

mrocklin reviewed May 31, 2019

View reviewed changes

pentschev changed the title ~~Fix device memory spilling with cuDF~~ [WIP] Fix device memory spilling with cuDF May 31, 2019

pentschev added 8 commits June 7, 2019 03:47

Merge remote-tracking branch 'upstream/branch-0.8' into fix-cudf-devi…

d1b9a6c

…ce-memory-spill

Add cudf spilling test with LocalCUDACluster

239490a

Improve LocalCUDACluster test for cudf spilling

156dbcd

Add Worker test for cudf spilling

58a1457

Mark LocalCUDACluster cudf spilling test asyncio

3093786

Rename CuPy spilling tests

f9d4174

Install cudf>=0.8 for CI

917148e

Install dask-cudf=0.8 for CI

da98927

mrocklin reviewed Jun 11, 2019

View reviewed changes

dask_cuda/tests/test_spill.py Outdated Show resolved Hide resolved

pentschev added 5 commits June 11, 2019 06:01

Increase cudf spilling test memory limits and data size

1377e3b

Avoid cudf->dask_cudf mapping in spilling test

a8bb30f

Resize cudf tests, compute sizes with map_partitions

526eccf

Avoid timeout/memory limit termination of cudf tests

53d01f8

Avoid eventual warnings/failures in spilling tests

0812fba

pentschev changed the title ~~[WIP] Fix device memory spilling with cuDF~~ [REVIEW] Fix device memory spilling with cuDF Jun 12, 2019

mrocklin reviewed Jun 13, 2019

View reviewed changes

pentschev added 3 commits June 13, 2019 08:20

Do not import cudf in _is_device_object

efe9523

Make spilling test assertions more consistent

c2122d7

Make _is_device_object recursive for list/tuple instances

13a3c09

Make is_device_object a generalized dispatcher

2c39520

kkraus14 approved these changes Jun 21, 2019

View reviewed changes

kkraus14 reviewed Jun 21, 2019

View reviewed changes

ci/gpu/build.sh Outdated Show resolved Hide resolved

kkraus14 reviewed Jun 21, 2019

View reviewed changes

dask_cuda/is_device_object.py Outdated Show resolved Hide resolved

pentschev and others added 2 commits June 21, 2019 10:05

Update ci/gpu/build.sh

1f3733e

Co-Authored-By: Keith Kraus <[email protected]>

Fix is_device_object_cudf_index function name

fdae239

Co-Authored-By: Keith Kraus <[email protected]>

pentschev added 2 commits June 21, 2019 01:29

Merge remote-tracking branch 'upstream/branch-0.8' into fix-cudf-devi…

432d050

…ce-memory-spill

Merge remote-tracking branch 'origin/fix-cudf-device-memory-spill' in…

ef2e9d0

…to fix-cudf-device-memory-spill

pentschev merged commit 5cffd41 into rapidsai:branch-0.8 Jun 21, 2019

pentschev mentioned this pull request Jul 25, 2019

Improve device memory spilling performance #98

Merged

VibhuJawa mentioned this pull request Jul 30, 2019

[FEA] Better CUDF/Nvstrings Spill over to Disk/Memory #99

Closed

pentschev deleted the fix-cudf-device-memory-spill branch September 9, 2019 08:48

pentschev mentioned this pull request Sep 12, 2019

Out of Memory Sort Fails even with Spill over #57

Closed

[REVIEW] Fix device memory spilling with cuDF #65

[REVIEW] Fix device memory spilling with cuDF #65

Conversation

pentschev commented May 31, 2019

mrocklin commented May 31, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pentschev commented May 31, 2019

pentschev commented May 31, 2019

VibhuJawa commented Jun 1, 2019 • edited Loading

pentschev commented Jun 1, 2019

VibhuJawa commented Jun 3, 2019 • edited Loading

pentschev commented Jun 3, 2019

VibhuJawa commented Jun 3, 2019 • edited Loading

beckernick commented Jun 4, 2019

pentschev commented Jun 11, 2019

pentschev commented Jun 12, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

galipremsagar commented Jun 14, 2019 • edited Loading

pentschev commented Jun 14, 2019

pentschev commented Jun 20, 2019

pentschev commented Jun 20, 2019

pentschev commented Jun 21, 2019

mrocklin commented Jun 21, 2019

pentschev commented Jun 21, 2019

VibhuJawa commented Jun 1, 2019 •

edited

Loading

VibhuJawa commented Jun 3, 2019 •

edited

Loading

VibhuJawa commented Jun 3, 2019 •

edited

Loading

galipremsagar commented Jun 14, 2019 •

edited

Loading