What is the reason for this problem during training？

[roseif]

Traceback (most recent call last):
File "train_dist.py", line 346, in
main()
File "train_dist.py", line 114, in main
mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
File "/home/user05/.conda/envs/open-mmlab1/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
while not spawn_context.join():
File "/home/user05/.conda/envs/open-mmlab1/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/home/user05/.conda/envs/open-mmlab1/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/home/user05/lai/PyTorch-Encoding/experiments/recognition/train_dist.py", line 328, in main_worker
train(epoch)
File "/home/user05/lai/PyTorch-Encoding/experiments/recognition/train_dist.py", line 256, in train
output = model(data)
File "/home/user05/.conda/envs/open-mmlab1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/user05/.conda/envs/open-mmlab1/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 464, in forward
self.reducer.prepare_for_backward([])
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel; (2) making sure all forward function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable). (prepare_for_backward at /opt/conda/conda-bld/pytorch_1579040055865/work/torch/csrc/distributed/c10d/reducer.cpp:514)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x47 (0x7f18ee0a3627 in /home/user05/.conda/envs/open-mmlab1/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10d::Reducer::prepare_for_backward(std::vector<at::Tensor, std::allocatorat::Tensor > const&) + 0x7b7 (0x7f191f8df557 in /home/user05/.conda/envs/open-mmlab1/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #2: + 0xa39bd1 (0x7f191f8cabd1 in /home/user05/.conda/envs/open-mmlab1/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #3: + 0x28ba06 (0x7f191f11ca06 in /home/user05/.conda/envs/open-mmlab1/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: _PyMethodDef_RawFastCallKeywords + 0x264 (0x5639e0ac0c34 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #5: _PyCFunction_FastCallKeywords + 0x21 (0x5639e0ac0d51 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #6: _PyEval_EvalFrameDefault + 0x52e6 (0x5639e0b2d4d6 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #7: _PyEval_EvalCodeWithName + 0x2f9 (0x5639e0a701b9 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #8: _PyFunction_FastCallDict + 0x1d5 (0x5639e0a712a5 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #9: _PyObject_Call_Prepend + 0x63 (0x5639e0a8fe33 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #10: PyObject_Call + 0x6e (0x5639e0a82a3e in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #11: _PyEval_EvalFrameDefault + 0x1f3a (0x5639e0b2a12a in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #12: _PyEval_EvalCodeWithName + 0x2f9 (0x5639e0a701b9 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #13: _PyFunction_FastCallDict + 0x1d5 (0x5639e0a712a5 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #14: _PyObject_Call_Prepend + 0x63 (0x5639e0a8fe33 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #15: + 0x16a2da (0x5639e0ac72da in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #16: _PyObject_FastCallKeywords + 0x49b (0x5639e0ac819b in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #17: _PyEval_EvalFrameDefault + 0x4a86 (0x5639e0b2cc76 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #18: _PyEval_EvalCodeWithName + 0xab8 (0x5639e0a70978 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #19: _PyFunction_FastCallKeywords + 0x387 (0x5639e0ac0437 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #20: _PyEval_EvalFrameDefault + 0x416 (0x5639e0b28606 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #21: _PyEval_EvalCodeWithName + 0xab8 (0x5639e0a70978 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #22: _PyFunction_FastCallDict + 0x1d5 (0x5639e0a712a5 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #23: _PyEval_EvalFrameDefault + 0x1f3a (0x5639e0b2a12a in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #24: _PyFunction_FastCallDict + 0x10b (0x5639e0a711db in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #25: _PyEval_EvalFrameDefault + 0x1f3a (0x5639e0b2a12a in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #26: _PyFunction_FastCallKeywords + 0xfb (0x5639e0ac01ab in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #27: _PyEval_EvalFrameDefault + 0x6a3 (0x5639e0b28893 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #28: _PyFunction_FastCallKeywords + 0xfb (0x5639e0ac01ab in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #29: _PyEval_EvalFrameDefault + 0x6a3 (0x5639e0b28893 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #30: _PyFunction_FastCallKeywords + 0xfb (0x5639e0ac01ab in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #31: _PyEval_EvalFrameDefault + 0x416 (0x5639e0b28606 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #32: _PyEval_EvalCodeWithName + 0x2f9 (0x5639e0a701b9 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #33: _PyFunction_FastCallKeywords + 0x387 (0x5639e0ac0437 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #34: _PyEval_EvalFrameDefault + 0x14eb (0x5639e0b296db in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #35: _PyEval_EvalCodeWithName + 0x2f9 (0x5639e0a701b9 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #36: PyEval_EvalCodeEx + 0x44 (0x5639e0a71094 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #37: PyEval_EvalCode + 0x1c (0x5639e0a710bc in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #38: + 0x22a9e4 (0x5639e0b879e4 in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #39: PyRun_StringFlags + 0x7d (0x5639e0b92d0d in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #40: PyRun_SimpleStringFlags + 0x3f (0x5639e0b92d6f in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #41: + 0x235e6d (0x5639e0b92e6d in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #42: _Py_UnixMain + 0x3c (0x5639e0b931ec in /home/user05/.conda/envs/open-mmlab1/bin/python)
frame #43: __libc_start_main + 0xf0 (0x7f192e217840 in /lib/x86_64-linux-gnu/libc.so.6)
frame #44: + 0x1daf7d (0x5639e0b37f7d in /home/user05/.conda/envs/open-mmlab1/bin/python)

[zhanghang1989]

Looks like a PyTorch issue. Sorry that I haven't experienced this error.

[XiaozhuLove]

我也遇到这个问题了，请问你解决了吗？谢谢。 @roseif