Join Horovod workers at the end of trainer.fit() to prevent race conditions following training

[tgaddair]

See: #1718 (comment)

[mergify]

This pull request is now in conflict... :(

[mergify]

This pull request is now in conflict... :(

[codecov]

Codecov Report

Merging #1786 into master will increase coverage by 0%.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #1786   +/-   ##
======================================
  Coverage      88%     88%           
======================================
  Files          69      69           
  Lines        4315    4316    +1     
======================================
+ Hits         3804    3805    +1     
  Misses        511     511           

[awaelchli]

I was requested for review but can only accept with low confidence.
I don't understand why the order of warning/import would cause a race condition.

Generally I would add a test for every bug fix to ensure it does not get re-introduced in the future, but not sure what to suggest here.

otherwise looks ok to me :)

[mergify]

Great job! =)

[awaelchli]

oh, I did not know it auto merges. Shouldn't there be a confirm button?

[Borda]

so be careful when you approve anything lol, there is a rule to merge all if tests pass and collect three approvals...

[williamFalcon]

lol. i moved those importa up bc they were failing flake8. but now they are at the bottom and flake passes?? @Borda

[Borda]

lol. i moved those importa up bc they were failing flake8. but now they are at the bottom and flake passes?? @Borda

yes, there was added ignore, the case was with new flake8 it started to check all warnings/error... before flake8 raised the most important one...

[tgaddair]

I was requested for review but can only accept with low confidence.
I don't understand why the order of warning/import would cause a race condition.

Generally I would add a test for every bug fix to ensure it does not get re-introduced in the future, but not sure what to suggest here.

otherwise looks ok to me :)

Just to clarify, the race condition fix line is here: https://github.com/PyTorchLightning/pytorch-lightning/pull/1786/files#diff-d673a98c14aa5cd59610bcc447178852R631. The other lines were introduced to make tests pass.

[williamFalcon]

awesome haha. I was commenting on our over-zealous flake8 haha :)