CuPy 7.2 breaks CI

[quasiben]

This is probably any issue with CuPy but wanted note here that we have seen breaking issues with CuPy 7.2

E   cupy.cuda.curand.CURANDError: CURAND_STATUS_INITIALIZATION_FAILED

cupy/cuda/curand.pyx:86: CURANDError

``` params = {'device_memory_limit': 1000000000.0, 'host_pause': None, 'host_spill': 0.0, 'host_target': 0.0, ...}

@pytest.mark.parametrize(
    "params",
    [
        {
            "device_memory_limit": 1e9,
            "memory_limit": 4e9,
            "host_target": 0.0,
            "host_spill": 0.0,
            "host_pause": None,
            "spills_to_disk": False,
        },
        {
            "device_memory_limit": 1e9,
            "memory_limit": 1e9,
            "host_target": 0.0,
            "host_spill": 0.0,
            "host_pause": None,
            "spills_to_disk": True,
        },
    ],
)
@pytest.mark.asyncio
async def test_cupy_cluster_device_spill(params):
    cupy = pytest.importorskip("cupy")
    with dask.config.set({"distributed.worker.memory.terminate": False}):
        async with LocalCUDACluster(
            1,
            scheduler_port=0,
            processes=True,
            silence_logs=False,
            dashboard_address=None,
            asynchronous=True,
            death_timeout=60,
            device_memory_limit=params["device_memory_limit"],
            memory_limit=params["memory_limit"],
            memory_target_fraction=params["host_target"],
            memory_spill_fraction=params["host_spill"],
            memory_pause_fraction=params["host_pause"],
        ) as cluster:
            async with Client(cluster, asynchronous=True) as client:

                rs = da.random.RandomState(RandomState=cupy.random.RandomState)

              x = rs.random(int(250e6), chunks=10e6)

dask_cuda/tests/test_spill.py:181:

---

/conda/envs/gdf/lib/python3.6/site-packages/dask/array/random.py:387: in random_sample
return self._wrap("random_sample", size=size, chunks=chunks)
/conda/envs/gdf/lib/python3.6/site-packages/dask/array/random.py:188: in _wrap
small_kwargs,
/conda/envs/gdf/lib/python3.6/site-packages/dask/array/random.py:453: in _apply_random
state = RandomState(state_data)
/conda/envs/gdf/lib/python3.6/site-packages/cupy/random/generator.py:51: in init
self._generator = curand.createGenerator(method)
cupy/cuda/curand.pyx:93: in cupy.cuda.curand.createGenerator
???
cupy/cuda/curand.pyx:97: in cupy.cuda.curand.createGenerator
???

---

???
E cupy.cuda.curand.CURANDError: CURAND_STATUS_INITIALIZATION_FAILED

cupy/cuda/curand.pyx:86: CURANDError

</details>

[jakirkham]

Does this go away when setting cupy.cuda.cub_enabled = False?

[pentschev]

Didn't we just pin a couple days ago an older version of CuPy because of that?

[quasiben]

Yes, we pinned. I was raising this issue to track that we are not working with 7.2

[pentschev]

Oh, I see. Sorry, I thought we were seeing that again, my mistake.

[jakirkham]

Added PR ( #248 ) to relax the pin since we've rebuilt the cupy package with a fix.