Large Memory Differences with DP vs. DDP accelerator #8826
Comments
Borda
added
distributed
|
Hey @anshulcgm when you run with DDP you need to divide the batch size by NUM_GPUS: if __name__ == "__main__":
# in DP, batch will be split by num gpus and then sent to gpu
run_trainer(accelerator = 'dp')
# in ddp, batch size is PER gpu, batches get processed by each worker individually.
BATCH_SIZE = BATCH_SIZE // NUM_GPUS
run_trainer(accelerator = 'ddp') |
awaelchli
added
question
|
Thanks for raising this issue Anshul, and thanks @awaelchli for the response. In my humble opinion, I think most people in optimization or machine learning would think of the batch size in SGD to be the number of total examples used for a single gradient update. this is also what how DDP treats the batch_size param in torch.utils.data.DataLoader. so i think this would be a good thing to consider changing in the next Lightning release : - ) |
that's unfortunately not how it is implemented in pytorch. there is a dataloader per process and the distributed sampler is partitioning the samples. the batch size is per process and the model forward() gets exactly a batch of that size. I think your argument can also easily be twisted around in favor of the current behavior in respect to scaling models to many GPUs. Also note that DDP and its variants is not an outlier in the behavor here. It's actually DP, which is the only parallel plugin that behaves the way you describe.
Can you elaborate on this part. I don't understand what you mean. |
|
This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team! |
🐛 Bug
I am running a training loop with a Transformer model with Pytorch Lightning and trying to use ddp as the accelerator. I run into CUDA OOM issues due to the large memory requirement of the multihead attention module, however I do not run into this issue when using DP as the accelerator. When tracking the GPU memory usage, DP runs through a batch using 25 GB of memory however DDP needs more than 45 GB.
To Reproduce
Console Output:
Save this script as
memory_error.pyand runpython memory_error.pyon any machine with 2+ GPUs with each GPU having >40 GB of memory. The GPU model that I am using is the NVIDIA A40 which has roughly 45 GB of memory.Expected behavior
Both dp and ddp should use similar amounts of memory to run this training loop, yet ddp uses significantly more memory.
Environment
conda,pip, source): condatorch.__config__.show():Additional context
The text was updated successfully, but these errors were encountered: