[distributed] NCCL Backend doesn't support torch.bool data type

[apsdehal]

🐛 Bug

In version 1.2.0, NCCL backend doesn't support torch.bool datatype. Broadcasting a tensor of this type throws error "RuntimeError: Unsupported data type for NCCL process group".

To Reproduce

Steps to reproduce the behavior:

Create a file test.py with following contents:

import torch
import argparse
from torch import distributed as dist


parser = argparse.ArgumentParser()
parser.add_argument("--local_rank", type=int)

args = parser.parse_args()

torch.distributed.init_process_group("nccl")

local_rank = args.local_rank

device = torch.device(local_rank)

if local_rank == 0:
    element = False
else:
    element = True


def broadcast_scalar(scalar, src=0, device="cpu"):
    scalar_tensor = torch.tensor(scalar).to(device)
    with torch.no_grad():
        scalar_tensor = dist.broadcast(scalar_tensor, src)
    return scalar_tensor.item()


broadcast_scalar(element, src=0, device=device)

Run it with following command:
python -u -m torch.distributed.launch --nproc_per_node 2 test.py

This has been tested on 2 GPUs.

Expected behavior

NCCL backend should support the bool datatype.
Current workaround fix: Change the datatype to int by doing .long() before broadcasting.

Environment

Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.1 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration:
GPU 0: Quadro GP100
GPU 1: Quadro GP100

Nvidia driver version: 410.79
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.16.3
[pip] torch==1.2.0
[pip] torchtext==0.3.1
[pip] torchvision==0.2.2
[conda] torch                     1.2.0                     <pip>
[conda] torchtext                 0.3.1                     <pip>
[conda] torchvision               0.2.2                     <pip>

[williamFalcon]

And you get a warning saying .uint8 isn't supported so we should switch to .bool... but then NCCL doesn't support it.

[soumith]

cc: @mrshenli @pietern we should fix this with upcast + transfer + downcast

[rohan-varma]

@soumith, I'm working on this bug. Could you explain what you mean by "upcast + transfer + downcast"? It appears that the error is coming from here: https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/ProcessGroupNCCL.cpp#L45-L60, and seems to happen because ncclDataType_t doesn't have a bool type

[soumith]

@rohan-varma i mean that we should cast the buffer from bool to uint8, then all reduce, then cast it on the other side to bool again

[pietern]

Things to keep in mind:

• Can only use uint8_t with up to 255 processes in the process group. We rely on every process contributing an integer equal to 1 if the equivalent boolean entry is set. With 256 processes we would overflow an 8-bit unsigned integer and get the wrong result. The change should either 1) assert that the process group size is small enough, or implement a separate code path that uses a 16-bit unsigned integer for larger process groups.
• Semantics of the different reduction ops (each of which could use SUM as the underlying reduction):
  • ReduceOp.SUM -- boolean OR, so the boolean output is output != 0
  • ReduceOp.PRODUCT -- boolean AND, so the boolean output is output == pg->size
  • ReduceOp.MIN -- boolean AND (see above)
  • ReduceOp.MAX -- boolean OR (see above)

[ekmb]

@rohan-varma Is it fixed?

[mrcslws]

I would be really happy to see this fixed. We have lots of modules that store masks in buffers. Because of this issue, these modules are forced to use float16 or float32 mask buffers rather than bool buffers.

[sajadn]

On pytorch 1.5.1 I still have problem with this. Is it going to be fixed?

[rohan-varma]

Hi @sajadn, this was landed ~1 month ago so it should be part of the next release, PT 1.7. Until then, you can try out the nightly build (see instructions at https://pytorch.org/) where this is fixed.