[DOC] "Single Cluster configuration" example doesn't work

[randerzander]

Using the latest conda nightlies, I'm attempting to use the LocalCUDACluster API with a DGX2, following the docpage.

ifconfig tells me I don't have a enp134s0f1 interface, so I used what looks like the relevant interface name on my box:

enp6s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500           
        inet 10.150.162.26  netmask 255.255.255.192  broadcast 10.150.162.63            
        inet6 fe80::5eff:35ff:fec1:75c4  prefixlen 64  scopeid 0x20<link>
        ether 5c:ff:35:c1:75:c4  txqueuelen 1000  (Ethernet)
        RX packets 466074749  bytes 155942292298 (155.9 GB)
        RX errors 0  dropped 6  overruns 0  frame 0               
        TX packets 4249216816  bytes 6392605233851 (6.3 TB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0x9d100000-9d17ffff 

The only other thing changed from the example was my worker directory:

from dask_cuda import LocalCUDACluster
from dask_cuda.initialize import initialize

# Configurations
protocol = "ucx"
interface = "enp6s0"
enable_tcp_over_ucx = True
enable_nvlink = True
enable_infiniband = False

initialize(
    create_cuda_context=True,
    enable_tcp_over_ucx=enable_tcp_over_ucx,
    enable_infiniband=enable_infiniband,
    enable_nvlink=enable_nvlink,
)

cluster = LocalCUDACluster(local_directory="/raid/rgelhausen/dask",
                        protocol=protocol,
                        interface=interface,
                        enable_tcp_over_ucx=enable_tcp_over_ucx,
                        enable_infiniband=enable_infiniband,
                        enable_nvlink=enable_nvlink,
                        rmm_pool="25GB",
                    )
client = Client(cluster)

Unfortunately, I get a very long repeating stack trace:

distributed.utils - ERROR - addresses should be strings or tuples, got None
Traceback (most recent call last):
  File "/home/rgelhausen/conda/envs/rapids-20200510_0117/lib/python3.7/site-packages/distributed/utils.py", line 664, in log_errors
    yield
  File "/home/rgelhausen/conda/envs/rapids-20200510_0117/lib/python3.7/site-packages/distributed/scheduler.py", line 2169, in remove_worker
    address = self.coerce_address(address)
  File "/home/rgelhausen/conda/envs/rapids-20200510_0117/lib/python3.7/site-packages/distributed/scheduler.py", line 4884, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None
distributed.core - ERROR - addresses should be strings or tuples, got None
Traceback (most recent call last):
  File "/home/rgelhausen/conda/envs/rapids-20200510_0117/lib/python3.7/site-packages/distributed/core.py", line 399, in handle_comm
    result = handler(comm, **msg)
  File "/home/rgelhausen/conda/envs/rapids-20200510_0117/lib/python3.7/site-packages/distributed/scheduler.py", line 2169, in remove_worker
    address = self.coerce_address(address)
  File "/home/rgelhausen/conda/envs/rapids-20200510_0117/lib/python3.7/site-packages/distributed/scheduler.py", line 4884, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None
distributed.utils - ERROR - addresses should be strings or tuples, got None
Traceback (most recent call last):
  File "/home/rgelhausen/conda/envs/rapids-20200510_0117/lib/python3.7/site-packages/distributed/utils.py", line 664, in log_errors
    yield
  File "/home/rgelhausen/conda/envs/rapids-20200510_0117/lib/python3.7/site-packages/distributed/scheduler.py", line 2169, in remove_worker
    address = self.coerce_address(address)
  File "/home/rgelhausen/conda/envs/rapids-20200510_0117/lib/python3.7/site-packages/distributed/scheduler.py", line 4884, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None

[pentschev]

Could you check that /raid/rgelhausen/dask is empty or try with some other empty directory? I've seen similar issues due to lingering files in that directory in the past.

[randerzander]

Unfortunately I get the same error with an empty worker directory.

[quasiben]

@randerzander I think it should be rmm_pool_size not rmm_pool and it's a mistake in the docs