[Fix] Fix iter bug when resuming checkpoint in distributed train

[FreyWang]

Motivation

As show in
https://github.com/open-mmlab/mmcv/blob/b4bfeb53c57f2c843cb5015f9c0a2d1689dba9c4/mmcv/runner/base_runner.py#L379 ,
cfg.gpu_ids is used to calculate current iteration when resuming checkpoint in epoch_based_runner and based_runner, but it is range(1) in distributed training in default, which will lead to a wrong iteration. Here fix it to range(world_size)

[codecov]

Codecov Report

Merging #866 (02ad416) into master (872e544) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #866   +/-   ##
=======================================
  Coverage   89.02%   89.02%           
=======================================
  Files         111      111           
  Lines        6043     6043           
  Branches      969      969           
=======================================
  Hits         5380     5380           
  Misses        467      467           
  Partials      196      196           

Flag	Coverage Δ
unittests	89.02% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 872e544...02ad416. Read the comment docs.

[Junjun2016]

Hi @FreyWang
Thanks for your contribution again.
It can resume the right number of iterations for epoch_based_runner which inherits resume from base_runner.
But for iter_based_runner which overwrites the resume function, maybe we need to adjust the max number of iterations.

[FreyWang]

Hi @FreyWang
Thanks for your contribution again.
It can resume the right number of iterations for epoch_based_runner which inherits resume from base_runner.
But for iter_based_runner which overwrites the resume function, maybe we need to adjust the max number of iterations.

I met the problem when using epoch_based_runner, It seems iter_based_runner override the function and have no probelm, but epoch_based_runner did not

[Junjun2016]

Hi @FreyWang
Thanks for your contribution again.
It can resume the right number of iterations for epoch_based_runner which inherits resume from base_runner.
But for iter_based_runner which overwrites the resume function, maybe we need to adjust the max number of iterations.

Do you have any idea to adjust the number of iterations for iter_based_runner?
We will discuss it in the next week.

[xvjiarui]

Should we add notes for resume with different number of GPUs somewhere?

Btw, I don't think resume with different batch size is a reasonable operator.

[FreyWang]

Hi @FreyWang
Thanks for your contribution again.
It can resume the right number of iterations for epoch_based_runner which inherits resume from base_runner.
But for iter_based_runner which overwrites the resume function, maybe we need to adjust the max number of iterations.

Do you have any idea to adjust the number of iterations for iter_based_runner?
We will discuss it in the next week.

Actually I agree that one resumed experiment should have same batch size with checkpoint.

[FreyWang]

Should we add notes for resume with different number of GPUs somewhere?

Btw, I don't think resume with different batch size is a reasonable operator.

I agree too, maybe batch size should be limited to be same