Bugfixes from Fairseq integration

[odp]

1. Divide grads in-place after end of accumulation steps
2. Constrain atomic_bsz by the user-provided limits before evaluating candidates. This fixes the buggy case of atomic_bsz_range=(B, B) used for achieving accumulation w/o batch scaling.
3. Additional Tensorboard logging
4. Use proper device for pinv

[codecov-commenter]

Codecov Report

Merging #129 (b5595d3) into master (3453440) will decrease coverage by 0.16%.
The diff coverage is 45.45%.

@@            Coverage Diff             @@
##           master     #129      +/-   ##
==========================================
- Coverage   61.38%   61.22%   -0.17%     
==========================================
  Files          32       32              
  Lines        2530     2535       +5     
  Branches      416      417       +1     
==========================================
- Hits         1553     1552       -1     
- Misses        896      901       +5     
- Partials       81       82       +1     

Impacted Files	Coverage Δ
adaptdl/adaptdl/torch/parallel.py	62.93% <0.00%> (-2.25%)	⬇️
adaptdl/adaptdl/torch/gradient_noise_scale.py	78.36% <60.00%> (ø)
adaptdl/adaptdl/goodput.py	97.56% <100.00%> (+0.01%)	⬆️
adaptdl/adaptdl/reducer.py	88.00% <0.00%> (-2.00%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.